Study guides · Intermediate

Feature Representations, Dimensionality Reduction, and Clustering

Examine how feature scales shape distance, then separate variance-preserving projection from clustering.

Ask Logeia about this articleTry the three questionsReview your answers

Updated

On this page
  1. Feature representation determines distance
  2. PCA preserves variance through projection
  3. k-means optimizes distances to centroids
  4. Unlabeled analysis still needs validation
  5. Ask Logeia about this articleCheck your understanding

Feature representation determines distance

The same person’s age has a different numerical scale when measured in years rather than days. Raw Euclidean distance can therefore be dominated by a feature’s units rather than its useful meaning. Standardization adjusts this influence but does not prove that every feature deserves equal importance. Arbitrary category numbers require care too: region code one need not be closer to code two than to code nine. Representation and distance are modeling choices that determine which structures an unsupervised method can find. They are not neutral formatting steps performed before the real analysis begins.

PCA preserves variance through projection

PCA finds orthogonal directions of high variance in centered data and projects observations onto a smaller set of those directions. If two-dimensional points lie near a diagonal line, one axis along that line may retain much of their variation. Discarding the orthogonal direction introduces reconstruction error. High variance is not equivalent to predictive or semantic importance; a small-variance feature can encode a crucial distinction. Fit the centering and projection on the training portion when evaluating a larger learning pipeline, then apply the same transformation to held-out data. Explained variance alone does not establish usefulness for the downstream task.

k-means optimizes distances to centroids

k-means assigns points to groups and seeks a small sum of squared distances to their centroids. For points zero, one, ten and eleven with two groups, a natural candidate is {0,1} and {10,11}, whose centroids are 0.5 and 10.5. Alternating assignment and centroid updates can reach different local solutions from different initializations. The distance objective favors certain shapes and scales; elongated or unequal-density groups may not match the desired interpretation. Cluster labels themselves have no intrinsic order or stable semantic meaning. A new run can permute their numeric names without changing the partition.

Unlabeled analysis still needs validation

Unsupervised does not mean unevaluated. Examine stability under changes in cluster count, distance, scaling and outlier treatment, as well as usefulness for the intended downstream task. An internal measure such as silhouette score quantifies separation under a chosen distance; it does not prove that a discovered group is a real user category. Reducing a visualization to two dimensions can alter distances and neighborhoods from the original space. Before giving clusters substantive names, check whether the representation or projection created the apparent separation. Interpretability depends on these checks and domain evidence, not only an appealing scatter plot.

Reading and practice are recorded separately in this browser, for each language.

Learn this in order

Prepare for AI systems · STEP 16

Choose a learning path