Handbook of Linear Algebra, 2nd Edition

Chapter 52

Least Squares Solution of Linear Systems

Per Christian Hansen

Technical University of Denmark

Hans Bruun Nielsen

Technical University of Denmark

52.1 Basic Concepts

(See Chapter 5 for additional information.)

Definitions:

Given a vector b ∈ ℝm and a matrix A ∈ℝm×n with m > n, the least squares problem is to find a vector x0 ∈ℝn that minimizes the Euclidean length of the difference between Ax and b:

$\mathbf{\text{Problem}}\mathbf{\text{LS}}:\text{Find}{\mathbf{x}}_0\text{satisfying}||\mathbf{b}-A{\mathbf{x}}_0{||}_2=min ||\mathbf{b}-A\mathbf{x}{||}_2.$

Such an x0 is called a least squares solution.

For any vector x the vector r = r(x) = b − Ax is the residual vector. The residual of a least squares solution is denoted by r0. The least squares problem is consistent if b ∈ range(A).

A basic solution, x0B, is a least squares solution ...