Backpropagation

Backpropagation hands each operation the gradient of the loss with respect to its own output, written $\bar z=\partial L/\partial z$ , and asks for the gradient with respect to its inputs. For $z=xy$ the answer is $\bar x=\bar z\,y$ and $\bar y=\bar z\,x$ — each input picks up the other one.

`x`	`y`	`upstream`	`dx`	`dy`
`3`	`4`	`1`	`4`	`3`
`3`	`4`	`10`	`40`	`30`
`-2`	`5`	`0.5`	`2.5`	`-1`
`7`	`0`	`1`	`0`	`7`

Two factors multiply together. The local derivative is what this operation contributes on its own: $\partial z/\partial x=y$ and $\partial z/\partial y=x$ . The chain rule multiplies that by whatever the rest of the network already worked out, the upstream $\bar z$ — so $\bar x=\bar z\cdot y$ . An upstream of $1$ therefore just reports the local derivatives, which is why the first row reads straight off the formula.

This is the entire pattern of a backward function: take upstream, multiply by the local derivative, return one gradient per input. Nothing here needs the value of $z$ — only the inputs that were cached from the forward pass.

Python

def multiply_backward(x: float, y: float, upstream: float) -> tuple[float, float]:
    """Backward pass of z = x * y.

    x, y:     the two scalar inputs of the forward multiply
    upstream: dL/dz, the gradient of the loss with respect to the output z
    Returns (dx, dy) = (dL/dx, dL/dy)
    """
    dx = upstream * y
    dy = upstream * x
    return dx, dy

Multiply backward with an upstream gradient

Add contributions at a fork