Write the objective explicitly
Regularization adds preferences about a solution to fitting the observed data, through an objective term or constraint. L2 regularization can add lambda times squared weight norm to mean squared loss, discouraging large coefficients. Lambda is a strength selected for a task and evaluation procedure, not a universally correct constant determined by notation. Summed loss and averaged loss give the same numerical lambda different relative influence. State the complete objective, including scaling conventions, before comparing parameter values across implementations or interpreting a change as stronger or weaker regularization.
Solve a scalar objective
For one observation y=2 and prediction w, use J(w)=(w−2) squared plus lambda w squared. Differentiating yields w=2/(1+lambda). The solutions are two, one, and one half for lambda zero, one, and three. This scalar example shrinks toward zero, but does not imply uniform shrinkage of every coefficient in a general multivariable problem. Choosing zero as the preferred center is itself a modeling decision. Whether an intercept receives the same penalty as other weights depends on the objective and implementation and should be checked explicitly.
Compare penalty geometry
L1 penalizes absolute coefficient magnitude and can produce exactly zero coefficients under suitable conditions. L2 penalizes squared magnitude smoothly and generally shrinks coefficients without setting them exactly to zero. Choice depends on interpretation, predictive goals, correlated features, and optimization. L1 does not guarantee recovery of precisely the truly important variables. Its ordinary derivative is undefined at zero, so the same derivative formula used for a smooth L2 term cannot be transferred unchanged. Match optimization and optimality conditions to the penalty’s geometry rather than treating all regularizers as interchangeable labels.
Check feature units
Multiplying a feature by one thousand lets the same prediction use a coefficient one thousand times smaller. Applying the same squared-coefficient penalty then changes the preference without changing the represented prediction. Feature units and scaling therefore interact directly with regularization. Fit standardization parameters on training data rather than using evaluation observations to compute means and variances. Otherwise preprocessing can leak information into the fitting process. A standardized coefficient’s magnitude also has a different interpretation from an effect expressed in the original feature units. Preserve that distinction in both model selection and reporting.
Distinguish penalties from update rules
For ordinary gradient descent, adding an L2 gradient introduces a multiplicative shrinkage update and can correspond to weight decay under a specified convention. Adaptive optimizers transform gradients coordinate by coordinate, so adding L2 to the loss and separately decaying weights are generally different operations. Factors of two and multiplication by the learning rate also depend on the implementation. Inspect the update rule rather than copying a numerical value based on a setting’s name. Early stopping and other mechanisms can additionally influence which solution is selected without an explicit penalty term in the written objective.
Select strength without contaminating evaluation
Fit each candidate strength on training data and compare candidates using validation data. Results used for selection are not an untouched final assessment. Preserve an appropriate final set or use a suitable cross-validation design. For the scalar example, calculate training error and penalty at lambda zero, one, and three. Comparing minimized objective values across different lambdas does not alone rank predictive quality because the objectives themselves differ. Connect this to bias and variance, but select using held-out performance and the constraints actually required by the application rather than one training statistic.