Study guides · Introductory

Gradient descent: test step size and convergence on a quadratic

Study the descent direction, learning rate, stopping criteria, and scaling through a small explicit calculation.

Ask Logeia about this articleTry the three questionsReview your answers

Updated

On this page
  1. Separate the direction from the distance
  2. Trace an explicit quadratic example
  3. Understand why scaling changes optimization
  4. Interpret stopping conditions conservatively
  5. Distinguish oscillation from divergence
  6. Ask Logeia about this articleCheck your understanding

Separate the direction from the distance

Gradient descent updates parameters in the direction opposite the gradient at the current point. The gradient describes local increase, so its negative is a useful candidate direction for decreasing the objective. The learning rate determines how far to travel. A valid local direction does not make every step length safe: moving too far can cross a valley and increase the loss. Read the update as two choices, a direction and a scale.

Trace an explicit quadratic example

For f(x)=x squared, start at x=4 and choose a learning rate of 0.1. The derivative is 2x, giving the first update 4 minus 0.1 times 8, or 3.2. The next value is 2.56. In general the next iterate equals(1-2η) times the current one. For this particular function, zero less than η less than one contracts the magnitude. At η=1 the sign flips without improvement; above one the magnitude grows. These bounds depend on the function.

python
x = 4.0
learning_rate = 0.1
for _ in range(3):
    x -= learning_rate * (2 * x)
    print(x)

Understand why scaling changes optimization

For f(x,y)=x squared plus 100y squared, the objective curves much more sharply in the y direction. A shared learning rate that is comfortable for x can make y oscillate or diverge. Feature scaling and learning-rate adjustment can therefore improve the geometry of the updates rather than merely hiding an implementation issue. More iterations cannot repair a step size that keeps increasing the error. Observe objective values and gradient magnitudes during the run.

Interpret stopping conditions conservatively

Common stopping rules include a small gradient norm, a small parameter update, negligible objective change, and a maximum iteration count. A tiny update alone may simply mean the learning rate is tiny. For a nonconvex objective, a stationary point need not be a global minimum. Stochastic gradients also introduce variation, so one upward step is not automatically a defect. State which convergence assumptions apply and distinguish reaching a computational stopping rule from proving that the best possible solution has been found.

Distinguish oscillation from divergence

On the same quadratic, a learning rate of 0.75 gives an update factor of minus 0.5. The iterate changes sign every time, but its magnitude halves and therefore converges. Oscillation in sign is not the same as divergence. A very small learning rate instead yields slow movement. Inspect objective values and parameter magnitudes along with the signs of updates so that a change in learning rate has an explanation beyond the appearance of a plotted curve.

Reading and practice are recorded separately in this browser, for each language.

Learn this in order

Prepare for AI systems · STEP 13

Choose a learning path