Statistics Simple Linear Regression

The Line We're Fitting

Simple linear regression describes a relationship between two numeric variables with a straight line: predicted y = intercept + slope * x. The intercept is the model's predicted y-value when x is zero, and the slope tells you how much the predicted y changes for each one-unit increase in x.

Least Squares: Finding the Best Line

Of all the possible straight lines you could draw through a scatter of points, least squares picks the one that minimizes the sum of the squared vertical distances between each actual data point and the line (the residuals). Squaring the residuals before adding them up means large misses are penalized much more heavily than small ones, and it keeps positive and negative residuals from simply canceling out.

def least_squares(x, y):
    n = len(x)
    mean_x = sum(x) / n
    mean_y = sum(y) / n
    numerator = sum((x[i] - mean_x) * (y[i] - mean_y) for i in range(n))
    denominator = sum((xi - mean_x) ** 2 for xi in x)
    slope = numerator / denominator
    intercept = mean_y - slope * mean_x
    return slope, intercept

distance_km = [2, 5, 8, 10, 14, 18]
delivery_min = [12, 20, 29, 35, 45, 58]

slope, intercept = least_squares(distance_km, delivery_min)
print("slope =", round(slope, 3), " intercept =", round(intercept, 3))

• The intercept (about 6.05 minutes here) is the model's predicted delivery time at a distance of 0 km - mostly reflecting fixed handling time before the driver even starts moving
• The slope (about 2.85 minutes per km here) is the predicted increase in delivery time for each additional kilometer of distance
• Predictions only make sense for distances similar to the range actually observed in the data (2 km to 18 km here)

How Well Does the Line Fit?

R-squared (the coefficient of determination) tells you what fraction of the variation in y is explained by the line's relationship with x. For simple linear regression, it happens to equal the square of the correlation coefficient. An R-squared of 0.80 means 80% of the variation in y is accounted for by its linear relationship with x, leaving 20% explained by other factors or randomness.

def least_squares(x, y):
    n = len(x)
    mean_x = sum(x) / n
    mean_y = sum(y) / n
    numerator = sum((x[i] - mean_x) * (y[i] - mean_y) for i in range(n))
    denominator = sum((xi - mean_x) ** 2 for xi in x)
    slope = numerator / denominator
    intercept = mean_y - slope * mean_x
    return slope, intercept

def r_squared(x, y, slope, intercept):
    n = len(x)
    mean_y = sum(y) / n
    predictions = [intercept + slope * xi for xi in x]
    ss_res = sum((y[i] - predictions[i]) ** 2 for i in range(n))
    ss_tot = sum((yi - mean_y) ** 2 for yi in y)
    return 1 - ss_res / ss_tot

distance_km = [2, 5, 8, 10, 14, 18]
delivery_min = [12, 20, 29, 35, 45, 58]

slope, intercept = least_squares(distance_km, delivery_min)
r2 = r_squared(distance_km, delivery_min, slope, intercept)
print("R-squared =", round(r2, 3))