Slope, before anyone says the word derivative

Two points and a division

Forget calculus for a moment. You have a function — anything that takes a number and returns a number — and you want to know whether it goes up or down near some point, and how steeply.

Pick your point, take a small step, and divide:

slope = ( f(x + h) - f(x) ) / h

That is the average rate of change over the step. Now shrink h. Take f(x) = x^2 at x = 3:

h = 1:      (16 - 9) / 1        = 7
h = 0.1:    (9.61 - 9) / 0.1    = 6.1
h = 0.01:   (9.0601 - 9) / 0.01 = 6.01
h = 0.001:  (9.006001 - 9)/.001 = 6.001

The numbers are converging on 6. That limit is the derivative of x² at 3, and the general formula 2x gives 2 × 3 = 6. No mystery: a derivative is the slope between two points, with the gap taken to zero.

Why it matters that it converges

The reason a derivative is useful is that it turns a question about a whole function into a single number at one point. Ask "if I nudge this weight up a little, does the loss go up or down, and how fast?" and the derivative answers it. Every training step is that question, asked millions of times.

The sign carries the direction and the magnitude carries the urgency. Derivative +6 means going right increases the value at six times the rate you go. Derivative 0 means flat, at least locally — a peak, a trough or a plateau.

Do not shrink h too far

Here is the practical wrinkle that a maths course will not tell you and a debugging session will. The formula gets more accurate as h shrinks, right up until floating-point arithmetic ruins it.

f(x + h) and f(x) are nearly equal for tiny h, so subtracting them cancels the leading digits and leaves you dividing noise by a very small number. With 64-bit floats the two effects balance around h ≈ 1e-8; go smaller and the answer gets worse.

def f(x): return x**2
x = 3.0
for h in (1e-4, 1e-8, 1e-12, 1e-16):
    print(h, (f(x+h) - f(x)) / h)
# 1e-04  6.000100000012054
# 1e-08  5.999999963900386
# 1e-12  6.000533403494046      <- getting worse
# 1e-16  0.0                    <- x + h rounds back to x

The last line is the whole lesson in one number. 3.0 + 1e-16 is exactly 3.0 in double precision, the numerator is zero, and the derivative comes back as zero. This is called catastrophic cancellation and it is why nobody computes gradients this way in production.

The version that is actually used for checking

A better finite difference uses a step in both directions:

central difference = ( f(x + h) - f(x - h) ) / (2h)

Its error shrinks with h² rather than h, so at h = 1e-5 it is typically accurate to seven or eight digits. This is the standard gradient check: compute the analytic gradient your code produces, compute the central difference, and compare. If they disagree by more than about 1e-5 relative, your backward pass has a bug.

It is worth knowing this exists because a wrong gradient does not crash. It trains — badly, mysteriously, in a way that looks like a hyperparameter problem for a week.

Derivatives you should be able to write down

Five cover most of what you meet, and it is worth being able to produce them without looking:

• f(x) = c (a constant): derivative 0. A flat line has no slope.
• f(x) = x^n: derivative n x^(n-1). So x² gives 2x, x³ gives 3x², and x gives 1.
• f(x) = e^x: derivative e^x. It is its own derivative, which is why e shows up everywhere in this subject.
• f(x) = ln(x): derivative 1/x. Steep near zero, nearly flat for large x.
• f(x) = 1/(1 + e^(-x)) (the sigmoid): derivative f(x) (1 - f(x)). Worth remembering as written, because it says the slope is computable from the output alone.

That last one has a consequence you will meet in this module. The sigmoid's output is between 0 and 1, so f(1 − f) is at most 0.25, at f = 0.5, and approaches zero at both ends. A sigmoid that has saturated has almost no gradient, and stacking several multiplies those small numbers together.

The two rules for combinations

• Sum: the derivative of f + g is the derivative of f plus the derivative of g. Slopes add.
• Constant multiple: the derivative of 3f is 3 times the derivative of f.

Together with the list above these let you differentiate most simple expressions. f(x) = 3x² + 5x − 7 has derivative 6x + 5; the constant vanishes because a constant has no slope.

The one remaining rule, for functions inside functions, is the chain rule, and it gets its own lesson because it is the entire mechanism of training a neural network.

The rule to keep

A derivative is a slope, computed by shrinking a gap. You can always check one numerically with a central difference at h = 1e-5, and doing so is the fastest way to find out whether a hand-written backward pass is right.