Separate the direction from the distance
Gradient descent updates parameters in the direction opposite the gradient at the current point. The gradient describes local increase, so its negative is a useful candidate direction for decreasing the objective. The learning rate determines how far to travel. A valid local direction does not make every step length safe: moving too far can cross a valley and increase the loss. Read the update as two choices, a direction and a scale.
Trace an explicit quadratic example
For f(x)=x squared, start at x=4 and choose a learning rate of 0.1. The derivative is 2x, giving the first update 4 minus 0.1 times 8, or 3.2. The next value is 2.56. In general the next iterate equals(1-2η) times the current one. For this particular function, zero less than η less than one contracts the magnitude. At η=1 the sign flips without improvement; above one the magnitude grows. These bounds depend on the function.
x = 4.0
learning_rate = 0.1
for _ in range(3):
x -= learning_rate * (2 * x)
print(x)Understand why scaling changes optimization
For f(x,y)=x squared plus 100y squared, the objective curves much more sharply in the y direction. A shared learning rate that is comfortable for x can make y oscillate or diverge. Feature scaling and learning-rate adjustment can therefore improve the geometry of the updates rather than merely hiding an implementation issue. More iterations cannot repair a step size that keeps increasing the error. Observe objective values and gradient magnitudes during the run.
Interpret stopping conditions conservatively
Common stopping rules include a small gradient norm, a small parameter update, negligible objective change, and a maximum iteration count. A tiny update alone may simply mean the learning rate is tiny. For a nonconvex objective, a stationary point need not be a global minimum. Stochastic gradients also introduce variation, so one upward step is not automatically a defect. State which convergence assumptions apply and distinguish reaching a computational stopping rule from proving that the best possible solution has been found.
Distinguish oscillation from divergence
On the same quadratic, a learning rate of 0.75 gives an update factor of minus 0.5. The iterate changes sign every time, but its magnitude halves and therefore converges. Oscillation in sign is not the same as divergence. A very small learning rate instead yields slow movement. Inspect objective values and parameter magnitudes along with the signs of updates so that a change in learning rate has an explanation beyond the appearance of a plotted curve.