Study guides · Intermediate

Understanding singular-value decomposition geometrically

Decompose a non-diagonal transformation into rotations and scaling, then verify reconstruction and a rank-one approximation.

Ask Logeia about this articleTry the three questionsReview your answers

Updated

On this page
  1. Choose directions before measuring stretch
  2. Build a non-diagonal example
  3. Check the input directions
  4. Keep one direction
  5. Understand non-uniqueness
  6. Distinguish projection from scaling
  7. Ask Logeia about this articleCheck your understanding

Choose directions before measuring stretch

How much a matrix stretches an input depends on its direction. In the singular value decomposition A=UΣV transpose, V identifies orthonormal input directions, Σ supplies nonnegative stretch factors, and U identifies orthonormal output directions. Read multiplication from right to left: change input coordinates, scale, then express the output in its own basis. Rectangular transformations are allowed, so the input and output spaces need not have equal dimension. We will use a two-dimensional example whose reconstruction can be checked with elementary multiplication rather than relying on a numerical decomposition as a black box. Keep the input and output dimensions explicit throughout the reconstruction check.

Build a non-diagonal example

Set U to the ninety-degree rotation with rows(0,−1) and(1,0), Σ to diag(3,1), and V to the forty-five-degree rotation with c=1/square root 2. Multiplication gives A with rows(c,−c) and(3c,3c). Applying A to v1=(c,c) produces(0,3); applying it to v2=(−c,c) produces(−1,0). Both inputs have unit length, but their outputs have lengths three and one. The example explains why diagonal entries, eigenvalues, and singular values should not be substituted for each other without identifying the transformation and directions involved.

Check the input directions

For this example, A transpose A has rows(5,4) and(4,5). Its eigenvalues along v1 and v2 are nine and one; their square roots give the singular values three and one. The matrix measures squared output norms for input directions, which explains the square root. However, explicitly forming A transpose A can square the condition number. This identity is a useful hand check, not a recommendation to implement every numerical SVD through normal equations. Distinguish an algebraic characterization from a stable general-purpose decomposition algorithm.

Keep one direction

Keeping only the largest singular component gives the rank-one approximation A1=3u1v1 transpose. Its first row is zero and its second row is(3c,3c). The discarded component has both spectral and Frobenius norm one. Input v1 is preserved exactly, while input v2 is mapped to zero instead of its original unit-length output. A small reconstruction error does not imply that every useful distinction survives compression. Evaluate downstream classification or retrieval separately, especially when an important low-frequency distinction lies in a direction with a small singular value.

Understand non-uniqueness

Simultaneously changing the sign of an input singular vector and its corresponding output vector leaves their outer product unchanged. Repeated singular values also permit different orthonormal bases within the same subspace. Two libraries can therefore return different-looking valid factors. Compare reconstruction, orthogonality, nonnegative singular values, and residual norms rather than demanding identical entries in every factor. Tolerances near zero should depend on matrix scale and numerical precision. A representation choice is not itself a different linear transformation, although unstable directions may still matter when interpreting individual components.

Distinguish projection from scaling

The input-space projector is P=v1v1 transpose, whose entries are all one half. It is symmetric and idempotent, unlike A1 itself. Projecting x=(2,0) produces(1,1), with residual(1,−1) orthogonal to the retained direction. Applying A afterward gives the same result as applying A1 directly. Verify those products and explain the difference between an input-space projection, a low-rank approximation to a transformation, and output scaling. The companion projection experiment checks these properties numerically, providing a bridge from matrix identities to observable vector calculations.

Reading and practice are recorded separately in this browser, for each language.

Learn this in order

Prepare for AI systems · STEP 9

Choose a learning path