Experiments and implementations · Intermediate

Testing gradient descent across step sizes

Trace a quadratic update to distinguish convergent oscillation, constant-amplitude alternation, and divergence as the learning rate changes.

Ask Logeia about this articleTry the three questionsReview your answers

Updated

On this page
  1. Fix the objective and update
  2. Predict both sign and magnitude
  3. Run the trajectories
  4. Inspect stopping criteria
  5. Extend to different curvatures
  6. Classify rather than cherry-pick
  7. Ask Logeia about this articleCheck your understanding

Fix the objective and update

Minimize f(x)=x squared divided by two. Its derivative is x and its unique minimum is at zero. Gradient descent updates x to(1−eta)x. Fix the initial value at four and change only eta. A differently scaled objective such as x squared has derivative two x, so the same numerical learning rate can behave differently. Write the full objective coefficient before comparing settings. This deterministic one-variable model checks an optimization mechanism; it is not GPU training or a measured result on a real machine-learning dataset.

Predict both sign and magnitude

When eta lies between zero and one, the multiplier is positive and smaller than one, so the sign stays fixed while magnitude shrinks. Between one and two, the sign alternates but magnitude still shrinks. At two, magnitude is constant; above two, it grows. These observations assume a nonzero initial point. Starting exactly at zero would hide the difference because every update stays there. Predict the loss x squared divided by two as well as parameter sign. Oscillation alone is not evidence of divergence, and visually smooth movement is not proof of rapid convergence.

Run the trajectories

Save the code as descent.py and run it with Python 3. Each learning rate receives twelve updates; the display includes the first five parameter values and final loss. At eta 0.25 the first updated value is three; at eta 1.5 it is minus two. Assertions compare final magnitude with the initial magnitude. The program uses no external files or network services apart from the script itself. If you change the objective or derivative, recalculate the expected trajectory rather than retaining assertions derived for a different update rule.

python
def trajectory(eta, steps=12):
    x = 4.0
    result = [x]
    for _ in range(steps):
        x = x - eta*x
        result.append(x)
    return result
for eta in [0.25, 1.5, 2.0, 2.1]:
    xs = trajectory(eta)
    print(eta, [round(x, 5) for x in xs[:5]], round(xs[-1]**2/2, 6))
assert abs(trajectory(0.25)[-1]) < 4
assert abs(trajectory(1.5)[-1]) < 4
assert abs(trajectory(2.0)[-1]) == 4
assert abs(trajectory(2.1)[-1]) > 4

Inspect stopping criteria

A tiny learning rate can make updates small while the point remains far from optimal. At eta equal to one millionth, the first movement from four is only four millionths despite a gradient of four. Stopping on update size alone would miss that. Combine gradient magnitude, objective behavior, and an iteration limit when defining termination. At eta two, loss remains constant while the parameter alternates between plus and minus four. Zero change in loss therefore does not prove arrival at the minimum. Explain what each stopping statistic can and cannot detect.

Extend to different curvatures

Extend the objective to(x squared plus one hundred y squared) divided by two. Its gradient is(x,100y), giving update multipliers 1−eta and 1 − 100 × eta. A learning rate suitable for x alone can diverge along y. Both directions contract here only when eta lies between zero and 0.02. This illustrates why scaling and preconditioning matter. Real stochastic learning additionally includes sample noise and possibly nonconvex geometry, so this exact quadratic boundary is not a universal learning-rate prescription. Use it to explain a mechanism, then inspect the assumptions of a more complicated training problem.

Classify rather than cherry-pick

Classify the four tested rates as same-sign convergence, alternating convergence, constant-amplitude alternation, and divergence. Justify each label with its multiplier and loss. Then compare eta 0.01 and 0.03 for the two-variable extension, where y can grow even while x appears well behaved. Completion means connecting learning rate with curvature and avoiding the claim that all oscillation is failure. Preserve the iteration count and terminal state instead of selecting only favorable early updates. Return to the convexity and smoothness guide when explaining which guarantees survive a change in objective or algorithm.

Reading and practice are recorded separately in this browser, for each language.

Learn this in order

Prepare for AI systems · STEP 13

Choose a learning path