Choose a function with a known derivative
Differentiate f(x)=x cubed minus two x plus two at x=1.5. Its analytic derivative is three x squared minus two, giving 4.75. The central finite difference estimates this value using[f(x+h)−f(x−h)] divided by 2h. This is a different computation from symbolic or automatic differentiation. Write down the function, analytic derivative, and approximation rule separately. A small example with a known answer lets you distinguish approximation error from an incorrectly implemented function before using finite differences to inspect a larger model or a complicated gradient implementation.
Prepare the calculation
Save the code as difference.py and execute it using Python 3’s ordinary float type without additional packages. Sweep h over several orders of magnitude, from one tenth down to ten to the minus twelve. Testing only one extremely small value cannot reveal whether approximation or rounding dominates. No external data, network access, or file mutation is involved. Record Python version, platform, numeric type, and the exact h list when comparing outputs. Floating-point differences across environments are a reason to use justified tolerances, not to demand identical printed decimal strings.
Compute estimates and errors
For each h, evaluate the function at two nearby points and divide the difference by 2h. Absolute error is the distance from 4.75. In exact arithmetic, this cubic’s central-difference truncation error equals h squared, so shrinking h initially helps. The assertion checks that at least one tested step provides a sufficiently accurate estimate. It deliberately does not require improvement at every smaller h or identical error across environments. Preserve the error column rather than only rounded estimates, since a rounded display can make distinct approximations appear equal even when their errors differ substantially.
def f(x):
return x*x*x - 2*x + 2
x = 1.5
exact = 3*x*x - 2
errors = []
for h in [1e-1, 1e-3, 1e-5, 1e-7, 1e-9, 1e-12]:
estimate = (f(x+h)-f(x-h))/(2*h)
error = abs(estimate-exact)
errors.append(error)
print(h, estimate, error)
assert min(errors) < 1e-6
print('analytic derivative', exact)
Explain the small-step limit
Very small steps make the two function values nearly equal. Subtraction loses significant digits, and division by a small 2h magnifies errors in those values. A rough model balances truncation on the order of h squared against roundoff on the order of machine precision divided by h. The best compromise also depends on function scale and higher derivatives. A step that works well for this cubic is not a universal setting for every variable, objective, or numeric type. Explain the competing effects rather than interpreting the smallest step as automatically the most accurate estimate.
Extend with care
For a multivariable function, perturb one coordinate while holding the others fixed to estimate a partial derivative. A directional difference instead compares with the gradient’s dot product with that direction. Nondifferentiable points require special care: the absolute-value function has central difference zero at the origin, but no ordinary derivative there. Stochastic operations introduce another source of variation. If two evaluations use different dropout masks or different minibatches, their difference can reflect randomness rather than the intended local change. Control those conditions before interpreting a gradient-check discrepancy.
Submit an interpretable error table
Submit a table of h, estimate, and absolute error, and identify the best tested step. Repeat at x=0 and x=3 with the corresponding analytic derivatives. Completion requires stating the known answer and explaining why improvement can stop as h shrinks. Retain the entire sweep rather than selecting only one favorable row; otherwise the step-size decision cannot be assessed. The numerical-stability guide explains related reformulations, while the gradient-descent experiment addresses a separate question: how a computed derivative and a learning rate jointly determine an update.