Regression

Definition

Regression is the supervised learning task of predicting a continuous number — a price, a temperature, a delivery time, a blood-pressure reading — from a set of input features. It is the direct counterpart to classification: where classification outputs a discrete label from a fixed set (spam or not-spam, cat or dog or horse), regression outputs a value that can fall anywhere on a scale, and two predictions can be arbitrarily close together. Asking "how many dollars?" is regression; asking "which of these five categories?" is classification. That single question — is the answer a number or a name? — decides which task you are solving and which metrics and models apply.

The naming trap worth clearing up immediately: logistic regression is not regression. It predicts a probability and is used to classify, and it has its own page. The word "regression" survives in its name for historical reasons, not because it predicts a continuous target.

How It Works

A regression model learns a function that maps input features to a numeric output. The simplest and most important case is linear regression, which fits a straight line y = wx + b: w is the slope (how much the prediction rises per unit of input) and b is the intercept (the prediction when the input is zero). Training means choosing the w and b that make the line pass as close as possible to the training points — and "as close as possible" has to be made precise before a computer can optimise it.

The standard definition of closeness is mean squared error (MSE): for each point, take the gap between the predicted value and the true value (the residual), square it, and average over all points. Squaring does two things — it makes positive and negative errors count the same, and it punishes large misses far harder than small ones. That second property is the one to internalise: an error of 10 contributes 100 to the sum, while an error of 1 contributes only 1. The big miss costs 100 times as much as the small one, not 10 times. This is exactly why regression is sensitive to outliers, a point we return to under Challenges.

A worked least-squares fit

Take five houses, with x = floor area in thousands of square feet and y = sale price in thousands of dollars:

For a straight line, the MSE-minimising slope and intercept have a closed form — no iteration needed. The slope is the covariance of x and y divided by the variance of x:

w = Σ(xᵢ − x̄)(yᵢ − ȳ) / Σ(xᵢ − x̄)², and then b = ȳ − w·x̄.

Here x̄ = 3 and ȳ = 250. The numerator sums to 500 and the denominator to 10, so w = 500 / 10 = 50 and b = 250 − 50·3 = 100. The fitted line is y = 50x + 100: every additional thousand square feet is worth about $50,000, and a hypothetical zero-area house would be priced at $100,000.

To judge the fit, run each x back through the line and compare to the true price. The predictions are 150, 200, 250, 300, 350, giving residuals of 0, 0, +10, −20, +10. Squaring and averaging the five residuals gives an MSE of 120, whose square root — the root mean squared error (RMSE), back in the original units — is about $10,950. The mean absolute error (MAE) is only 8 ($8,000), and the gap between RMSE and MAE is a fingerprint of that one −20 residual: the house at x = 4 alone contributes 400 of the 600 total squared error, two-thirds of it from a single point.

Reading R-squared

MSE is in squared units and has no natural scale, so on its own it does not say whether a fit is good. R² (the coefficient of determination) fixes that by comparing the model against the dumbest possible baseline — always predicting the mean. It is 1 − SS_res / SS_tot, where SS_res is the model's total squared error (600 here) and SS_tot is the total squared error of predicting ȳ = 250 for every house (25,600 here). So R² = 1 − 600 / 25600 = 0.977: the line explains about 97.7% of the variance in price. R² = 1 is a perfect fit, R² = 0 means the model is no better than the mean, and R² can even go negative when a model is worse than that baseline — a genuinely useful red flag that plain MSE hides.

Linear regression has this tidy closed-form solution, but most models do not. When the function is a polynomial, a neural network, or a gradient-boosted ensemble, the same MSE objective is minimised iteratively with gradient descent, which nudges the parameters downhill on the error surface until it flattens out. The loss function being minimised is the part that stays the same; only the search procedure changes.

Types

Unlike many glossary terms, regression does carry a few genuinely distinct model families, worth separating because they make different assumptions about the shape of the relationship:

• Linear regression fits a straight line (or, with several inputs, a flat hyperplane) and assumes the target changes at a constant rate per feature. It is the most interpretable choice — each coefficient reads directly as "units of output per unit of input" — and includes the regularised variants ridge and lasso, which shrink coefficients to curb overfitting.
• Polynomial regression adds powers of the inputs (x², x³, …) as extra features so the fitted curve can bend. It is still linear in its parameters, so the same least-squares machinery applies, but high-degree polynomials wiggle to chase noise and overfit badly.
• Multiple (multivariate) regression is any of the above with more than one input feature — the everyday case, where price depends on area and bedrooms and location at once.

A deliberate non-member: logistic regression predicts a probability and solves a classification problem, so it belongs on its own page, not in this list. If you find yourself inventing category names to fill this section, there is no taxonomy — but linear, polynomial, and multiple are real distinctions practitioners name in everyday work.

Real-World Applications

Regression is the quiet default whenever the answer is a number rather than a label. Zillow's Zestimate is a regression model that predicts a continuous dollar value for tens of millions of homes from features like square footage, location, and recent nearby sales. Insurers and lenders run regression to price premiums and estimate expected loss on a loan, where the output is a monetary amount. Retailers and manufacturers use it for demand forecasting — predicting next week's unit sales for each SKU so warehouses stock the right quantity — and utilities forecast electricity load in megawatts to schedule generation. In medicine, regression models predict continuous clinical targets such as estimated glomerular filtration rate (a kidney-function number) or a patient's projected length of stay. In each case the choice of regression over classification is forced by the question: nobody wants "expensive vs cheap" for a house when they can have a dollar figure.

Key Concepts

The vocabulary that recurs across every regression model:

• Residual — the signed gap between a true value and its prediction (yᵢ − ŷᵢ). The whole fit is built to make these small, and plotting them is the first diagnostic when something looks wrong.
• MSE / RMSE / MAE — the three standard error metrics. MSE and its square root RMSE penalise large errors quadratically; MAE averages absolute errors and so shrugs off outliers. Reporting RMSE alongside MAE tells you at a glance whether a few big misses are inflating the average.
• R² — variance explained, on a fixed 0-to-1-ish scale, so results are comparable across datasets in a way raw MSE is not.
• Extrapolation — predicting outside the range of the training inputs. The house line above says a 10,000 sq ft mansion is worth $600,000, but no house that size was in the data, so that number is a guess the model is in no position to make.

Challenges

Outliers dominate the loss. Because MSE squares errors, one mislabeled or genuinely extreme point can bend the whole line toward itself — as the single −20 residual above supplied two-thirds of the total squared error. When outliers are noise rather than signal, switching the objective to MAE (or a Huber loss, which is quadratic for small errors and linear for large ones) stops a handful of points from hijacking the fit.

Overfitting versus underfitting. A line that is too rigid (underfitting) misses real curvature; a degree-12 polynomial that threads every training point exactly (overfitting) has memorised the noise and predicts wildly between points. The fix is to hold out data the model never trains on and watch the error on that — training error always keeps falling with complexity, so it cannot be the thing you optimise.

Multicollinearity. When two input features move together — floor area and number of rooms, say — the model cannot tell which one is driving the target, so the individual coefficients become unstable and their signs can even flip with a small change in the data. The overall prediction may still be fine, but any interpretation of a single coefficient is no longer trustworthy.

Getting the units and scale right. RMSE and MAE are in the units of the target, so a model predicting prices in dollars and one predicting them in thousands of dollars will report errors that differ by 1000× while being the identical model. Reporting a bare error number without its units, or comparing R² across two different targets as if it were an accuracy score, is a common way to draw a wrong conclusion from a correct model.

Code Example

The closed-form least-squares fit from above, computed from scratch so every quantity is visible:

# Five houses: x = size (1000s of sq ft), y = sale price ($1000s)
x = [1, 2, 3, 4, 5]
y = [150, 200, 260, 280, 360]
n = len(x)

# Least-squares fit of y = w*x + b (closed form)
x_mean = sum(x) / n
y_mean = sum(y) / n
w = sum((xi - x_mean) * (yi - y_mean) for xi, yi in zip(x, y)) \
    / sum((xi - x_mean) ** 2 for xi in x)
b = y_mean - w * x_mean
print(f"fitted line: y = {w:.0f}*x + {b:.0f}")

# Predictions and errors
preds = [w * xi + b for xi in x]
resid = [yi - pi for yi, pi in zip(y, preds)]
print("residuals:", resid)

mse = sum(r ** 2 for r in resid) / n
rmse = mse ** 0.5
mae = sum(abs(r) for r in resid) / n
print(f"MSE  = {mse:.1f}")
print(f"RMSE = {rmse:.2f}")
print(f"MAE  = {mae:.1f}")

ss_res = sum(r ** 2 for r in resid)
ss_tot = sum((yi - y_mean) ** 2 for yi in y)
r2 = 1 - ss_res / ss_tot
print(f"R^2  = {r2:.3f}  (SS_res={ss_res}, SS_tot={ss_tot})")

Running it prints:

fitted line: y = 50*x + 100
residuals: [0.0, 0.0, 10.0, -20.0, 10.0]
MSE  = 120.0
RMSE = 10.95
MAE  = 8.0
R^2  = 0.977  (SS_res=600.0, SS_tot=25600.0)

In practice you would call LinearRegression().fit(X, y) from scikit-learn and get the same w = 50 and b = 100 in one line — but the twenty lines above are the entire idea: measure error by squaring residuals, then pick the slope and intercept that make that sum as small as it can be.