Layers compose smaller functions
A neural network composes smaller functions, often weighted sums and nonlinear activations. Forward computation evaluates intermediate values until it produces a prediction. Stacking only linear maps still yields a linear map, so nonlinearities matter for expressiveness. More layers do not automatically make the learned rule appropriate for a task. Start by specifying inputs, tensor shapes, parameter sharing, outputs and the loss definition. A computation graph makes these dependencies explicit. It describes a mathematical function and its evaluation, rather than providing an explanation that the system necessarily understands a concept in a human sense.
Follow the chain rule for one weight
Let prediction z=wx and loss L be one half of (z-y) squared. With x=2, w=3 and y=5, the prediction is six and loss is 0.5. The derivative of loss with respect to z is one, while the derivative of z with respect to w is two, giving dL/dw=2. Backpropagation follows dependencies in reverse and combines local derivatives using the chain rule. When an intermediate value contributes through several paths, the contributions add. Following only one path would miss part of the derivative of a shared parameter.
Differentiation and optimization are separate
With gradient descent and learning rate 0.1, the new weight is 3 minus 0.1 times two, or 2.8. The new prediction is 5.6 and loss is 0.18, below the earlier 0.5. This is a worked calculation, not evidence that every learning rate reduces loss. Automatic differentiation computes derivatives of the specified graph; an optimizer chooses an update from them. If gradients are accumulated across batches, clearing and averaging them at the intended points matters. Otherwise, the update magnitude can differ from the mathematical objective the programmer thought was being optimized.
A computable gradient still has limitations
Reverse-mode differentiation needs intermediate information from the forward pass, creating a memory-versus-recomputation trade-off. Products of very small or large derivatives can lead to vanishing or exploding gradients. Activations also have specific behavior, such as the usual zero derivative of ReLU on its negative region. Small hand calculations and finite-difference checks can help validate an implementation, with care near nondifferentiable points and floating-point limits. Correct gradients establish that the chosen calculation is differentiated correctly; they do not repair data leakage, an unsuitable objective or a misleading evaluation procedure.