Some links on this page are affiliate links: if you buy through them we may earn a commission, at no extra cost to you.
Ordinary least squares (OLS) projects the observed response vector onto the column space of the design matrix. The fitted values are that nearest point; the residual is the perpendicular gap. This geometric view explains the normal equations, the hat matrix, and why fitted values can remain unique even when regression coefficients are not.
What “geometry” means in linear regression
There are several useful pictures of regression. In a scatterplot, simple regression fits a line through observations. In coefficient space, the sum of squared errors is a bowl-shaped surface. But the most useful picture for multiple regression is in observation space: each variable is a vector with one entry per observation, and OLS projects the response vector onto the subspace allowed by the model. This is the standard geometric interpretation of least squares (regression geometry lecture notes).
If there are n observations, the vectors live in ℝn. That space is usually too high-dimensional to draw. A line or plane in a diagram is an analogy; the underlying projection is not generally taking place in the two-dimensional axes of a scatterplot.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
How the design matrix defines the model space
For simple linear regression with an intercept, write the model as yi = β0 + β1xi + εi. With n observations, its design matrix is:
#1 Best Overall
X = [1, x], with one row per observation. Its first column is the all-ones vector 1, representing the intercept; its second is the observed predictor vector x. Every possible fitted vector is Xβ = β01 + β1x. These vectors form the span of the two columns in ℝn, assuming the columns are independent.
For multiple regression, X has a column for each model term: an intercept, numerical predictors, indicator columns for categories, polynomial terms, or interactions. Its column space, written C(X), is the set of all fitted-value vectors the model can produce. Changing the coefficient coordinates may leave this space—and therefore all possible predictions—unchanged (design-matrix and observation-space explanation).
Why least squares is a nearest-point problem
OLS chooses coefficients to minimize the residual sum of squares:
RSS(β) = ||y − Xβ||²
The response y is a point in observation space. Candidate fitted vectors Xβ are restricted to C(X). The minimizing fitted vector is the point in that space closest to y in ordinary squared Euclidean distance:
ŷ = projC(X)(y)
The residual e = y − ŷ is the vector from the fitted point to the observed response. At the nearest point, this residual is perpendicular to the whole model space. This is the central result behind the geometric account of OLS (OLS projection and ANOVA geometry).
Keep the fitted vector and the coefficient vector distinct. ŷ describes the projection in observation space; β̂ gives coordinates used to express that point through the columns of X. If columns are dependent, different coefficient vectors can express the same fitted vector.
Residual orthogonality gives the normal equations
At the minimum, the residual must have zero component in every direction the model can move. The allowable directions are the columns of X, so:
Rank #2
Xᵀe = 0, or equivalently, Xᵀ(y − Xβ̂) = 0.
Rearranging gives the normal equations:
XᵀXβ̂ = Xᵀy
“Normal” here means perpendicular, not normally distributed. If the columns of X are linearly independent, XTX is invertible and the solution can be written β̂ = (XTX)−1XTy. This is an important derivation, but explicitly forming that inverse is often not the preferred numerical way to fit a model; QR or SVD methods are generally more stable, especially for poorly conditioned designs (linear algebra and least-squares course notes).
With an intercept, orthogonality to the first column means 1Te = 0, so residuals sum to zero. Orthogonality to each included predictor column means its dot product with the residual is zero. This is an algebraic property of the fitted OLS model—not proof that residuals are independent, pattern-free, or uncorrelated with every omitted or nonlinear function of the predictors (linear models and normal equations).
How the projection relates to a familiar fitted line
In simple regression with an intercept, the fitted line passes through the centroid (x̄, ȳ). The slope and intercept are:
β̂₁ = Σ(xᵢ − x̄)(yᵢ − ȳ) / Σ(xᵢ − x̄)²β̂₀ = ȳ − β̂₁x̄
The zero-sum residual condition explains why the fitted responses average to ȳ; combining that fact with the fitted line’s form gives the centroid result (simple-regression geometry and the centroid).
In a scatterplot, residuals are drawn as vertical differences between observed and fitted y-values at each x. The residual vector’s perpendicularity is a different statement: in ℝn, its dot product with every design-matrix column is zero. The first is a visual representation of individual errors in the plot; the second is the vector condition characterizing the least-squares solution.
Multiple regression and partial effects
With several predictors, OLS still projects y onto C(X), now spanned by all model columns. The coefficients describe a joint projection, not separate simple-regression lines. One way to understand a particular coefficient is partial regression: remove from that predictor the component explained by the other predictors, and relate what remains to the corresponding part of the response after accounting for those same predictors. The coefficient therefore concerns variation left after the other included columns have been projected out (partial-regression interpretation).
This also helps explain omitted-variable bias in a linear projection. If an omitted variable’s effect has a component aligned with an included predictor after accounting for other included variables, that component can be absorbed into the included predictor’s coefficient. This describes the geometry of association; it does not establish a causal effect, which requires additional design and identification assumptions (nested-model and regression geometry).
PC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11Outdated Drivers Are Slowing You Down
One free scan finds every outdated or missing driver and matches the right update for your exact hardware.Free scan · exact hardware matchThe hat matrix and leverage
When X has full column rank, the projection can be written using the hat matrix:
H = X(XᵀX)⁻¹Xᵀŷ = Hy, e = (I − H)y
H is symmetric (HT = H) and idempotent (H2 = H). Symmetry is characteristic of an orthogonal projection; idempotence means projecting a vector that is already in the model space does not move it again. When the design is rank deficient, the projection is instead written H = XX+, where X+ is the Moore–Penrose pseudoinverse (projection matrices and pseudoinverses).
The diagonal entry hii measures observation i’s leverage: how its predictor configuration relates to the others and how strongly it can affect its own fitted value. High leverage does not by itself mean an observation is erroneous or influential. Influence depends on the response residual as well as leverage.
How orthogonal projections explain R² and ANOVA
With an intercept, the centered response decomposes into fitted variation and residual variation:
Free tools Windows power users keep installed
One-click scans. No signup required.
y − ȳ1 = (ŷ − ȳ1) + e
The two components are orthogonal, so the Pythagorean theorem gives:
TSS = SSR + SSE
Here TSS is the total sum of squared deviations from the mean, SSR is the squared length of the centered fitted component, and SSE is the residual sum of squares. Thus, for this intercept-centered OLS fit, R2 = 1 − SSE/TSS = SSR/TSS. In simple regression with an intercept, it is also the squared correlation between x and y (sum-of-squares decomposition).
Rank #4
For nested models, the smaller model’s column space is contained in the larger model’s. The larger projection can fit additional directions. The squared length of the additional fitted component is the extra sum of squares; comparing it with residual variation under the larger model leads to partial F-tests and the familiar ANOVA decomposition. The geometry explains what the comparison measures, while inferential conclusions still depend on the model’s statistical assumptions.
These identities need qualification. Without an intercept, the constant vector is not necessarily in the model space, so residuals need not sum to zero and the usual centered sum-of-squares decomposition may not hold. Some out-of-sample or no-intercept definitions of R2 can be negative. Neither a high R2 nor a geometric decomposition establishes causality, valid uncertainty estimates, good predictions, or well-behaved residual assumptions.
Recommended Free Tools
Rank deficiency, collinearity, and stable computation
If one design-matrix column is an exact linear combination of others, X is rank deficient. Then XTX is singular and coefficient vectors may not be unique. The fitted vector remains the unique nearest point in the fixed column space, but the data cannot identify a unique allocation of that fit among dependent columns. Near-dependence is different: coefficients may be highly sensitive to small data changes, even when predictions are comparatively stable. When the number of columns is at least the number of observations, multiple coefficient vectors can also produce equally good fits unless further restrictions are imposed.
QR decomposition
Write X = QR, where the columns of Q form an orthonormal basis for the column space. Then the projection is QQTy. This basis makes the perpendicular geometry explicit and avoids directly forming XTX.
Singular-value decomposition
The SVD writes X = UΣVT. The relevant left-singular vectors describe the column space. SVD is useful for detecting rank deficiency and near-dependence, computing pseudoinverse solutions, and examining small singular values. QR and SVD are numerical ways to compute the least-squares projection, not different statistical models; rank decisions in practice depend on numerical tolerances (course coverage of rank, QR, and SVD).
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Where ordinary projection geometry changes
The simple projection formula applies to OLS under squared Euclidean loss. Other methods alter the loss, the distance measure, or both:
Do these 3 things before closing this tab:
1Scan for outdated or missing drivers - takes under a minute2Repair Windows errors before they cause bigger problems3Fix the driver behind crashes, sound loss and screen glitches| Method | Geometric qualification |
|---|---|
| Ordinary least squares | Orthogonal projection onto C(X) under the usual Euclidean inner product. |
| Weighted least squares | Uses a weighted distance and corresponding weighted notion of perpendicularity. |
| Generalized least squares | Uses a covariance-adjusted metric rather than ordinary Euclidean distance. |
| Ridge regression | An L² penalty shrinks coefficients; the fitted-value smoother is generally not an idempotent orthogonal projection. |
| Lasso | An L¹ penalty changes the optimization geometry and does not give the ordinary OLS projection. |
| Robust regression or L¹ regression | Uses a different loss, so ordinary residual orthogonality need not characterize its solution. |
| Instrumental variables | Its estimating and projection interpretation differs from OLS projection of the response onto the regressor column space. |
The broad takeaway is not that all regression is projection in the same sense; it is that OLS has a particularly clean projection interpretation (OLS geometry and alternative penalties).
Best Value
A hand-checkable example
Take three observations with an intercept and one predictor:
X = [[1,1], [1,2], [1,3]], y = [1,2,2]ᵀ
The fitted line is ŷ = 2/3 + (1/2)x, giving:
ŷ = [7/6, 5/3, 13/6]ᵀe = y − ŷ = [−1/6, 1/3, −1/6]ᵀ
Check both design directions: the residual entries sum to zero, and their dot product with the predictor values is also zero:
1ᵀe = −1/6 + 1/3 − 1/6 = 0xᵀe = 1(−1/6) + 2(1/3) + 3(−1/6) = 0
So the residual is perpendicular to both columns of X and therefore to every fitted vector the model can produce.
Verify the geometry in Python or R
Use a least-squares solver rather than explicitly computing (XTX)−1. Floating-point arithmetic may return tiny values near zero instead of exact zeros.
Python with NumPy
import numpy as np
X = np.column_stack([
np.ones(3),
np.array([1.0, 2.0, 3.0])
])
y = np.array([1.0, 2.0, 2.0])
beta_hat, residuals, rank, singular_values = np.linalg.lstsq(X, y, rcond=None)
y_hat = X @ beta_hat
e = y - y_hat
print("beta_hat:", beta_hat)
print("y_hat:", y_hat)
print("residual:", e)
print("X.T @ residual:", X.T @ e)The last line should be approximately [0, 0].
R
x <- c(1, 2, 3)
y <- c(1, 2, 2)
fit <- lm(y ~ x)
coef(fit)
fitted(fit)
resid(fit)
crossprod(model.matrix(fit), resid(fit))The final expression checks the normal equations for the model matrix used by lm.
The Tool Desk
Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →What the geometric picture does—and does not—tell you
- The columns of X define the allowable fitted-value space.
- OLS chooses the projection of y onto that space.
- The residual is perpendicular to every included model direction.
- Coefficients are coordinates used to express the fitted vector; rank conditions determine whether those coordinates are unique.
This explains how OLS fits a linear model. It does not, by itself, establish unbiasedness, efficient estimation, valid standard errors, independence, normality, causal validity, or predictive performance.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
