When a matrix preserves a direction
For a square matrix A, a nonzero vector v is an eigenvector when Av equals λv. The transformation keeps that vector on its original line, although its length or sign may change. For A equal to diag(3,1), horizontal vectors grow by three and vertical vectors keep their length. A general vector combines both directions and usually changes direction. Having eigenvalues does not guarantee enough independent eigenvectors for diagonalization. Always distinguish the equation defining one eigenpair from the stronger assertion that a complete eigenbasis exists.
Projection makes the residual orthogonal
The orthogonal projection of x onto a unit vector u is u times the inner product of u and x. With u=(1,0) and x=(2,3), the projection is (2,0), leaving residual (0,3). Its inner product with u is zero. For a matrix X with independent columns, the least-squares residual is orthogonal to every column of X. This does not mean the residual vanishes: it means no movement inside the chosen subspace can reduce squared error further. Dependent columns require care because the usual normal-equation inverse may not exist.
Use SVD for rectangular matrices
A singular-value decomposition writes A as UΣV-transpose. V describes orthogonal input directions, Σ scales them and U describes orthogonal output directions. The singular values of diag(3,1) are three and one. Keeping only the first produces the rank-one approximation diag(3,0), discarding the second component. Unlike an eigenvalue decomposition, SVD also applies to rectangular matrices. Singular values are nonnegative, so they do not preserve the sign of a negative eigenvalue. State the dimensions of U, Σ and V before multiplying them; reduced and full decompositions use different shapes.
Decide what a discarded direction means
Applying SVD after centering a data matrix connects the decomposition to principal component analysis. Retaining high-variance directions does not necessarily retain the directions best at predicting a target. Different measurement units can change the result, and tiny singular values make inversion sensitive to perturbations. Check reconstruction and residual norms instead of trusting the factor names alone. Floating-point calculations also need scale-appropriate tolerances for orthogonality and eigen-equation checks. A discarded component is a modeling decision with a measurable reconstruction cost, not automatic evidence that the component contains only noise.