Skip to content
LoveToCalc

Theorem — what does that mean?

Least squares solution

A system with more equations than unknowns usually has no solution. The least squares solution is the vector that misses by the least.

In Swedish: Minsta kvadratlösning (normalekvationerna)

Least squares solution

The relation

(x̂₁, x̂₂)=(a₁·ba₁·a₂a₂·ba₂·a₂a₁·a₁a₁·a₂a₁·a₂a₂·a₂, a₁·a₁a₁·ba₁·a₂a₂·ba₁·a₁a₁·a₂a₁·a₂a₂·a₂)

When Ax = b has no solution, the vector x̂ that makes ‖b − Ax‖ smallest solves the normal equations AᵀA x̂ = Aᵀb. For two unknowns the card solves them with Cramer’s rule.

Solved for x̂

x̂=(a₁·ba₁·a₂a₂·ba₂·a₂a₁·a₁a₁·a₂a₁·a₂a₂·a₂, a₁·a₁a₁·ba₁·a₂a₂·ba₁·a₁a₁·a₂a₁·a₂a₂·a₂)

x̂=(1, 0.5)

x̂ equals the vector (((the determinant with rows (6, 6) and (13, 14)) divided by (the determinant with rows (3, 6) and (6, 14))), ((the determinant with rows (3, 6) and (6, 13)) divided by (the determinant with rows (3, 6) and (6, 14)))), equals the vector ((6 divided by 6), (3 divided by 6)), equals the vector (1, 0.5)

Working3 steps

Practice questions

You are given

  • a₁·b=-15
  • a₁·a₂=8
  • a₂·b=-9
  • a₂·a₂=5
  • a₁·a₁=14
Your answer: The least squares solution x̂

The normal equations

The system Ax = b, with A an m×n matrix and m larger than n, usually has no solution. The least squares solution x̂ is the vector that makes the length ‖b − Ax‖ smallest. It is the solution of the normal equations

For two unknowns, call the columns of A a₁ and a₂. Then AᵀA has the rows (a₁·a₁, a₁·a₂) and (a₁·a₂, a₂·a₂), and Aᵀb is (a₁·b, a₂·b). The card above takes these five numbers and solves the 2×2 system with Cramer’s rule. Forming them from A and b is the part you do first: each is a dot product of two columns.

Holds when the columns of A are linearly independent, which is when det AᵀA is not 0. Otherwise there are infinitely many least squares solutions.

Worked example

A has the rows (1, 1), (1, 2) and (1, 3), and b = (2, 1, 3). The columns are a₁ = (1, 1, 1) and a₂ = (1, 2, 3).

The normal equations are 3x̂₁ + 6x̂₂ = 6 and 6x̂₁ + 14x̂₂ = 13, with det AᵀA = 3·14 − 6·6 = 6.

Check: Ax̂ = (1.5, 2, 2.5), so b − Ax̂ = (0.5, −1, 0.5). This residual is orthogonal to both columns: 0.5 − 1 + 0.5 = 0 and 0.5 − 2 + 1.5 = 0. Its squared length is 0.25 + 1 + 0.25 = 1.5.

These are the points (1, 2), (2, 1) and (3, 3), and x̂ = (1, 0.5) is the line y = 1 + 0.5x through them in the least squares sense. The page on the least squares line gets the same line from the closed formulas for slope and intercept; those are these normal equations solved once and for all for the columns (1, …, 1) and the x-values.

A second example: orthogonal columns

A has the columns a₁ = (1, 1, 0) and a₂ = (1, −1, 0), and b = (3, 1, 2). Here a₁·a₂ = 0, so the two equations separate: x̂₁ = (a₁·b)/(a₁·a₁) = 4/2 = 2 and x̂₂ = (a₂·b)/(a₂·a₂) = 2/2 = 1. Then Ax̂ = (3, 1, 0), which is the orthogonal projection of b onto the plane spanned by the columns, and the residual (0, 0, 2) is perpendicular to it.

That is the general picture: Ax̂ is the point of the column space closest to b.

When the columns are dependent, as for A with the rows (1, 2), (2, 4) and (3, 6), det AᵀA = 14·56 − 28² = 0, and the card says so.

Common mistakes

Frequently asked questions

Why are they called normal equations? Aᵀ(b − Ax̂) = 0 says that the residual is normal, that is perpendicular, to every column of A. That is exactly what makes x̂ the closest point.

What if there are more than two unknowns? The normal equations are the same, AᵀA x̂ = Aᵀb, with an n×n system. Solve it by Gaussian elimination or with the matrix calculator.

Is the least squares solution an exact solution when there is one? Yes. If Ax = b has a solution, the residual is 0 and x̂ is that solution (unique when the columns are independent).

Looking for another letter? The formula solved for each one
Solved for each letter
Solve forRearrangedWith the numbersComes to
x̂₁a₁·ba₁·a₂a₂·ba₂·a₂a₁·a₁a₁·a₂a₁·a₂a₂·a₂661314366141
x̂₂a₁·a₁a₁·ba₁·a₂a₂·ba₁·a₁a₁·a₂a₁·a₂a₂·a₂36613366140.5

Last reviewed November 7, 2026 ·How pages are reviewed

Calculate it in practice

Linear algebraAll formulas

Put this formula on your site

Copy the code into your page where the formula card should appear. It loads from lovetocalc.com, so it stays current on its own — all we ask is that the credit line comes with it.