Derivatives for Learning

The derivative $f'(x)$ is the slope of $f$ at $x$ . Estimate it with the central difference: evaluate $f$ a tiny step on each side of $x$ and take the slope between the two points.

f'(x) \approx \frac{f(x+h) - f(x-h)}{2h}

`f`	`x`	`h`	result	exact
$x^2$	`3`	`1e-5`	`6.000`	$6$
$\sin x$	`0`	`1e-5`	`1.000`	$1$
$e^x$	`1`	`1e-5`	`2.718`	$e$
$x^3$	`1`	`0.5`	`3.250`	$3$ (a coarse $h$ shows the error)
constant $5$	`2`	`1e-5`	`0`	$0$

Why the symmetric form: the one-sided $(f(x+h)-f(x))/h$ has an error proportional to $h$ , while the central form cancels the even-order terms and its error is proportional to $h^2$ . Keep $h$ small but not tiny: below about $10^{-8}$ the subtraction $f(x+h)-f(x-h)$ loses its leading digits to rounding. This estimate is the yardstick that gradient checking later compares analytic gradients against.

Python

def derivative(f, x: float, h: float = 1e-5) -> float:
    """Central-difference estimate of f'(x): (f(x + h) - f(x - h)) / (2h).

    f: a function float -> float
    x: the point at which to estimate the slope
    h: half-width of the step (default 1e-5)
    Returns a float.
    """
    return (f(x + h) - f(x - h)) / (2 * h)

Numerical derivative

Numerical gradient