The normal equations
The system Ax = b, with A an m×n matrix and m larger than n, usually has no solution. The least squares
solution x̂ is the vector that makes the length ‖b − Ax‖ smallest. It is the solution of the normal equations
For two unknowns, call the columns of A a₁ and a₂. Then AᵀA has the rows (a₁·a₁, a₁·a₂) and (a₁·a₂, a₂·a₂), and
Aᵀb is (a₁·b, a₂·b). The card above takes these five numbers and solves the 2×2 system with
Cramer’s rule. Forming them from A and b is the part you
do first: each is a dot product of two columns.
Holds when the columns of A are linearly independent, which is when det AᵀA is not 0. Otherwise there are
infinitely many least squares solutions.
Worked example
A has the rows (1, 1), (1, 2) and (1, 3), and b = (2, 1, 3). The columns are a₁ = (1, 1, 1) and a₂ = (1, 2, 3).
- a₁·a₁ = 3, a₁·a₂ = 1 + 2 + 3 = 6, a₂·a₂ = 1 + 4 + 9 = 14
- a₁·b = 2 + 1 + 3 = 6, a₂·b = 2 + 2 + 9 = 13
The normal equations are 3x̂₁ + 6x̂₂ = 6 and 6x̂₁ + 14x̂₂ = 13, with det AᵀA = 3·14 − 6·6 = 6.
- x̂₁ = (6·14 − 13·6) / 6 = 6/6 = 1
- x̂₂ = (3·13 − 6·6) / 6 = 3/6 = 0.5
Check: Ax̂ = (1.5, 2, 2.5), so b − Ax̂ = (0.5, −1, 0.5). This residual is orthogonal to both columns:
0.5 − 1 + 0.5 = 0 and 0.5 − 2 + 1.5 = 0. Its squared length is 0.25 + 1 + 0.25 = 1.5.
These are the points (1, 2), (2, 1) and (3, 3), and x̂ = (1, 0.5) is the line y = 1 + 0.5x through them in the
least squares sense. The page on the
least squares line gets the same line from the closed
formulas for slope and intercept; those are these normal equations solved once and for all for the columns
(1, …, 1) and the x-values.
A second example: orthogonal columns
A has the columns a₁ = (1, 1, 0) and a₂ = (1, −1, 0), and b = (3, 1, 2). Here a₁·a₂ = 0, so the two equations
separate: x̂₁ = (a₁·b)/(a₁·a₁) = 4/2 = 2 and x̂₂ = (a₂·b)/(a₂·a₂) = 2/2 = 1. Then Ax̂ = (3, 1, 0), which is the
orthogonal projection of b onto the plane
spanned by the columns, and the residual (0, 0, 2) is perpendicular to it.
That is the general picture: Ax̂ is the point of the column space closest to b.
When the columns are dependent, as for A with the rows (1, 2), (2, 4) and (3, 6), det AᵀA = 14·56 − 28² = 0, and the
card says so.
Common mistakes
- Solving some of the equations exactly. Taking the first two equations of the example, x₁ + x₂ = 2 and
x₁ + 2x₂ = 1, gives (3, −1), and the third equation then misses by 3, a squared error of 9 against 1.5.
- AAᵀ instead of AᵀA. AᵀA is n×n, with n the number of unknowns. AAᵀ is m×m and does not belong here.
- Forgetting the right-hand side’s transpose. The right-hand side is Aᵀb, the dot products of the columns
with b, not b itself.
- Expecting a unique answer with dependent columns. Then det AᵀA = 0 and the normal equations have
infinitely many solutions.
Frequently asked questions
Why are they called normal equations?
Aᵀ(b − Ax̂) = 0 says that the residual is normal, that is perpendicular, to every column of A. That is
exactly what makes x̂ the closest point.
What if there are more than two unknowns?
The normal equations are the same, AᵀA x̂ = Aᵀb, with an n×n system. Solve it by
Gaussian elimination or with the
matrix calculator.
Is the least squares solution an exact solution when there is one?
Yes. If Ax = b has a solution, the residual is 0 and x̂ is that solution (unique when the columns are
independent).