Machine LearningLinear Regression

Linear Regression Explained

y=mx+b, cost function (MSE), gradient descent, R² score, multivariable regression — with Python implementation.

April 11, 20243 min readRishabh Singh
Gradient descent animation: line fitting to data points iteratively
Gradient descent: adjusting the line's slope and intercept step by step until the error is minimized.

The Foundation: y = mx + b

Linear regression starts from the same equation you learned in school. Given data points (years, income), the goal is to find the line that best fits them — the one that minimizes prediction error across all points. This is the workhorse behind causal forecasting — predicting an outcome from its drivers.

ŷ = θ₀ + θ₁x    (hypothesis function)

θ₀ is the intercept (y-axis offset), θ₁ is the slope. The model learns these parameters from data.

Scatter plot of Canadian per capita income vs years with best-fit regression line
Canadian per capita income vs years — the regression line captures the trend through all data points.

The Cost Function

How do you measure how "wrong" the line is? The cost function quantifies total prediction error across all training examples. Linear regression uses Mean Squared Error (MSE) — also called Least Squared Error.

J(θ) = (1/2m) Σ (ŷᵢ − yᵢ)²
Cost function J(theta) — mean squared error across all training examples
Cost function J(θ): average squared distance between predicted and actual values.

Squaring errors serves two purposes: negative errors don't cancel positive ones, and large errors are penalized more severely than small ones.

Gradient Descent

Gradient descent finds the parameters θ that minimize J(θ). Starting from a random point, it computes the gradient (slope of the cost function surface) and takes a step downhill. Repeat until convergence.

θⱼ := θⱼ − α × ∂J(θ)/∂θⱼ

α is the learning rate — too large and it overshoots, too small and it converges slowly.

Gradient descent on cost function surface — bowl shape with arrow descending to minimum
Gradient descent: each step moves toward the minimum of the cost function bowl.
Gradient descent animation on 2D cost surface converging to global minimum
Gradient descent converging on the 2D cost surface — each iteration reduces the error.

Evaluation Metrics

Mean Absolute Error (MAE)

MAE = (1/m) Σ |ŷᵢ − yᵢ|

Average absolute difference. Interpretable in original units. Less sensitive to outliers than MSE.

MAE formula — mean absolute error for regression evaluation
MAE: average of absolute prediction errors — interpretable in original data units.

R² Score (Coefficient of Determination)

R² = 1 − SS_res / SS_tot

Explains the proportion of variance in the target explained by the model. R² = 1 means perfect fit; R² = 0 means the model does no better than predicting the mean.

R² score visualization showing explained vs unexplained variance
R² score: what fraction of target variance does the model explain?

Multivariable Linear Regression

Real datasets have multiple features. Multivariable linear regression extends the hypothesis to many input dimensions:

ŷ = θ₀ + θ₁x₁ + θ₂x₂ + … + θₙxₙ

With 2 features, the model fits a plane instead of a line. With 3+ features, it fits a hyperplane — no longer visualizable, but the math is identical (the same hyperplane concept that SVMs use as a decision boundary). Whatever the dimension, the model outputs point estimates — to quantify how uncertain they are, wrap them in confidence and prediction intervals.

Multivariable linear regression: best-fit plane through 3D data points
Multivariable regression with 2 features: the model fits a plane rather than a line.
Multivariable linear regression Python code with sklearn
Multivariable linear regression implementation with sklearn.
Linear regression prediction output and R² score result
Model output: predictions and R² score on test data.
"The goal is always the same: find the parameters that minimize the cost function. The dimension just changes."

Implementation Notebooks

Full implementations available on GitHub Gist:

Key Takeaways

  • Hypothesis: ŷ = θ₀ + θ₁x — a line (or hyperplane) parameterized by θ.
  • Cost function (MSE): measures total squared error — what gradient descent minimizes.
  • Gradient descent: iteratively adjusts θ downhill on the cost surface.
  • R² score: how much variance the model explains — closer to 1 is better.
  • Multivariable: same algorithm, extra θs — fits a plane or hyperplane instead of a line.
Back to BlogRead on Medium