Distinguish conditioning and stability
Conditioning describes how a problem responds to changes in its input; numerical stability concerns the behavior of an algorithm under rounding and related computational errors. Two formulas for the same mathematical quantity can produce different finite-precision results. Conversely, a stable method cannot recover information absent from uncertain input to a sensitive problem. Specify which input digits are trustworthy and how output error is measured. Neither fast execution nor a small iteration count establishes numerical stability. These are separate properties that require different observations and, when possible, an error analysis.
Observe cancellation
For small x, consider square root of one plus x minus one. The expression is mathematically valid, but subtracting nearby quantities can destroy significant digits. At x=10 to the minus sixteen, ordinary binary 64 arithmetic may round one plus x to one and produce zero. The exact mathematical result is not zero. Trace where rounding occurs before the final subtraction rather than blaming every small number. Floating-point values form a discrete set, and relative accuracy of a small difference can be poor even when the intermediate larger quantities are individually well represented.
Reformulate before increasing precision
Multiplying by the conjugate gives x divided by(square root of one plus x plus one). Near zero, this avoids subtracting two nearly equal quantities and yields about five times ten to the minus seventeen for the example. Check the domain and denominator before applying the identity. Algebraic reformulation is not a universal guarantee, but it can be preferable to simply increasing precision. Specialized functions such as expm1 and log1p address related small-difference calculations. Select a method suited to the scale and structure of the expression, then test its behavior against a reference.
Check residual and solution error
A small residual does not necessarily imply a small solution error. Let A be diag(1,0.001), the true solution(1,1), and the approximation(1,2). The residual b−A xhat is(0,−0.001), but the solution error is(0,1). The matrix suppresses the second direction, hiding a substantial solution discrepancy in a small equation discrepancy. In a real problem where the exact answer is unknown, combine scaled residuals with conditioning estimates and knowledge of measurement error. Report what each diagnostic measures instead of referring to every small number as accuracy.
Choose meaningful tolerances
Relative error becomes difficult to interpret near a zero reference. A mixed test such as absolute difference no greater than atol plus rtol times reference magnitude can combine absolute and relative tolerances. Those tolerances depend on units and accuracy requirements. Converting seconds to milliseconds also requires converting an absolute tolerance. A permissive threshold is not a substitute for validating a model or implementation. Record why the chosen tolerance is appropriate, which value acts as the reference, and whether the comparison is intended to catch rounding, approximation error, or a materially different computation.
Use reference calculations and adversarial inputs
Test ordinary inputs as well as large scale differences, boundary cases, and sign changes. Compare both square-root formulas at ten to the minus four, minus eight, and minus sixteen, preferably against a higher-precision reference. Record numeric type, environment, and tolerances. For linear systems, construct a known solution first and form b=A x so that residual and solution error can both be measured. After investigating algorithmic stability, input reliability and problem conditioning still remain separate concerns. A useful numerical report explains all three layers rather than compressing them into a single pass or fail label.