Neural Networks Math

A developer who can read $Z = XW + b$ and $\bar W = X^T \bar Z$ and immediately write the code — and explain why it is right — understands neural networks. This topic builds exactly that skill, one formula at a time.

The approach: formula → shapes → code. Each card opens with a piece of notation, states the shape contract (samples are rows: $X$ is $B \times D$, $W$ is $D \times H$, $b$ is $H$, every gradient has the shape of the thing it differentiates), shows a worked example, and then asks you to implement it. Hints first unpack the indices behind the symbol — what $\sum_i$ ranges over, which axis collapses — and then give the exact NumPy / Eigen idiom. You do not re-implement matrix multiplication; you learn to express the math the way production code does, and you hand-implement the things that are the lesson: gradient descent, reverse-mode differentiation, and the training loop.

Progression. Notation & Shapes (what scalars, vectors, dot products and matrix equations mean) → Notation to Array Idioms (sums over an axis, transpose, Hadamard and outer products, broadcasting a bias, one-hot, argmax) → Derivatives for Learning (numerical derivatives and the chain rule, which later double as a gradient-checking oracle) → Activations & Losses (sigmoid, ReLU, stable softmax, MSE, cross-entropy from logits) → Layers & Forward (a neuron, a dense layer on a batch, a two-layer forward pass with a cache) → Gradient Descent (one step, learning rates, linear and logistic regression gradients) → Backpropagation (upstream gradients, forks, local backward functions, the dense-layer identities derived entry by entry, softmax-cross-entropy's $P - Y$, wiring a two-layer backward pass, gradient checking) → Training a Network (a mini-batch step, symmetry breaking, XOR end to end).

Languages. Python with NumPy and C++ with Eigen, side by side: the same exercise reads X @ W + b in one and X * W / rowwise() += b in the other, so the mapping from symbol to code is seen twice.