Experiments and implementations · Intermediate

Exploring the geometry of projection matrices

Project a two-dimensional vector onto a line and verify orthogonal residuals, idempotence, and the squared-norm decomposition.

Ask Logeia about this articleTry the three questionsReview your answers

Updated

On this page
  1. Choose a unit direction
  2. Write the projection matrix
  3. Verify three geometric properties
  4. Test a non-unit direction
  5. Connect orthogonality to least squares
  6. Bridge to low-rank approximation
  7. Ask Logeia about this articleCheck your understanding

Choose a unit direction

Let the retained line have unit direction u=(0.6,0.8), and let the input be x=(3,−1). The direction has length one because its squared components sum to one. Orthogonal projection is p=u times the dot product of u and x. That dot product is one, giving p=(0.6,0.8) and residual r=(2.4,−1.8). Sketch the three vectors before running code. Projection selects the component in a specified subspace; it is not merely an operation that shrinks every coordinate independently or discards whichever coordinate looks numerically small.

Write the projection matrix

The matrix form is P=u u transpose, with rows(0.36,0.48) and(0.48,0.64). Multiplying P by x must agree with the dot-product construction. An orthogonal projector under the ordinary real inner product is symmetric and idempotent: its transpose equals itself and its square equals itself. Either property alone is not the full characterization. Building P from a unit direction supplies both in this example. Verify the matrix entries by hand before treating the program as evidence; the hand calculation provides an independent reference for the numerical assertions.

Verify three geometric properties

Save the code as projection.py and execute it with Python 3; no additional packages are required. It computes the projection, residual, and a second projection of the projected vector. Assertions check idempotence, residual orthogonality, and the Pythagorean squared-norm decomposition. Floating-point tolerances account for rounding rather than demanding textual equality with zero. The program changes no external service or data file. Expected squared norms are approximately ten, one, and nine, so the printed result can also be checked by elementary arithmetic independently of the assertions.

python
import math
u = (0.6, 0.8)
x = (3.0, -1.0)
dot = lambda a,b: sum(v*w for v,w in zip(a,b))
project = lambda a: tuple(dot(u,a)*v for v in u)
p = project(x)
r = tuple(a-b for a,b in zip(x,p))
pp = project(p)
print('projection', p, 'residual', r, 'dot', dot(u,r))
assert math.isclose(dot(u,u), 1.0)
assert all(math.isclose(a,b,abs_tol=1e-12) for a,b in zip(p,pp))
assert abs(dot(u,r)) < 1e-12
assert math.isclose(dot(x,x), dot(p,p)+dot(r,r))
print('squared norms', dot(x,x), dot(p,p), dot(r,r))

Test a non-unit direction

Replace u by(3,4) without changing the formula and the result is no longer the correct projection, despite describing the same line. The unit-length assumption has been violated. For any nonzero direction v, use v times its dot product with x divided by v’s squared norm. Dividing by twenty-five restores the original result here. Modify the program and verify invariance when the direction is multiplied by a nonzero constant. Reject the zero vector explicitly: its denominator is zero and it does not define a line direction for this calculation.

Connect orthogonality to least squares

Every point on the line has form t u. The squared distance from x is minimized when t equals the dot product of u and x, which is one here. Testing t equal to zero, one, and two gives squared distances ten, nine, and ten. Residual orthogonality expresses this least-squares condition. The retained component has length one while the residual has length three; they measure different quantities. Minimum distance depends on the chosen inner product and norm. A weighted-error problem requires its own geometry rather than an automatic reuse of the unweighted formula.

Bridge to low-rank approximation

Completion means explaining from both equations and outputs why p lies on the line, the residual is perpendicular, and a second projection changes nothing. Repeat with input(−1,2). For several orthonormal directions collected as columns of Q, the projector generalizes to Q Q transpose. The same expression is not valid for arbitrary nonorthogonal columns without adjustment. Projecting onto selected right-singular-vector directions connects this experiment to SVD and low-rank approximation. Keep the input-space projection distinct from approximating a transformation matrix or from evaluating whether discarded information matters for a downstream task.

Reading and practice are recorded separately in this browser, for each language.

Learn this in order

Prepare for AI systems · STEP 8

Choose a learning path