Separate input and output directions
A real matrix admits a singular-value decomposition A=UΣV transpose. The columns of V describe orthogonal input directions, U describes output directions, and nonnegative entries of Σ describe stretching between them. Input and output dimensions can differ, so SVD applies to rectangular matrices rather than only square ones. State whether the full or reduced form is used and check the dimensions of every factor. Similar-looking symbols are not enough to make a multiplication valid. Following how input coordinates are rotated, scaled, and mapped to output coordinates is more reliable than memorizing the order of letters without understanding their roles.
Measure stretching in a diagonal example
For A=diag(3,1), horizontal inputs stretch by three and vertical inputs by one, so the singular values are three and one. Among inputs with Euclidean length one, the largest output length is three. Replacing the first diagonal entry with minus three leaves the singular values unchanged. A direction factor accounts for reversal while stretching magnitudes remain nonnegative. Eigenvalues, by contrast, would include minus three. This simple example shows why signed eigenvalues cannot be substituted for singular values. Treat magnitude and orientation as different parts of the decomposition rather than assuming that every spectral quantity describes the same aspect of the transformation.
Connect singular values to A transpose A
The matrix A transpose A is square, symmetric, and positive semidefinite. Its eigenvalues are the squares of A’s singular values. For diag(3,1), it is diag(9,1), yielding singular values three and one. Right singular directions connect to these eigenvectors. This mathematical relationship does not mean explicitly forming A transpose A is always a good numerical algorithm; doing so can worsen conditioning. Separate a defining identity from a stable implementation. A library’s SVD routine should be selected and configured according to dimensions, required factors, and accuracy rather than replaced casually with a home-built normal-matrix eigenvalue calculation.
Read rank and approximation from the spectrum
The number of nonzero singular values is the matrix rank. Setting the smaller singular value of diag(3,1) to zero gives the rank-one approximation diag(3,0). Inputs along the discarded direction can no longer be reconstructed. Truncated SVD retaining the largest singular values yields an optimal low-rank approximation for the specified spectral or Frobenius norm. That geometric optimum does not prove that discarded information is irrelevant to a prediction or business task. A low-variance direction can distinguish rare but important cases. Evaluate reconstruction error separately from downstream usefulness instead of interpreting a small singular value as automatic evidence of meaningless noise.
Interpret small values relative to scale
Finite precision can turn an exact zero singular value into a tiny positive result. Conversely, a fixed absolute cutoff can discard a small but meaningful direction when the data scale changes. Numerical rank estimation should consider the largest singular value, matrix dimensions, arithmetic precision, and input uncertainty. Repeated singular values can also make individual singular vectors nonunique: another orthonormal basis of the same subspace may be equally valid. Different factor columns do not by themselves establish an error. Check reconstruction, orthogonality, and the relevant subspace before concluding that two decompositions disagree mathematically. The numerical representation should be interpreted through these invariants.
Probe a small singular direction
For diag(3,0.1), input (0,1) has unit length but produces an output of length 0.1. The largest singular value, three, does not describe that particular input direction. Maximum amplification and sensitivity of weakly represented directions are different questions. Inspect the relevant part of the spectrum together with the input direction instead of reporting only the largest value and assuming it explains every observation.