Layers & Forward

A neuron weighs its inputs, adds a bias, and squashes the result: $a=\sigma(w\cdot x+b)$ , where $\sigma(z)=1/(1+e^{-z})$ is the sigmoid. Everything else in a network is this operation repeated and stacked.

`w`	`x`	`b`	result
`[1, 1]`	`[1, -1]`	`0`	`0.5000`
`[0.5, -1]`	`[2, 1]`	`1`	`0.7311`
`[0, 0, 0]`	`[5, -2, 7]`	`2`	`0.8808`
`[1]`	`[1]`	`-1000`	`0.0000` (no overflow)

Two steps, and the shapes tell you which is which. The weighted sum $w\cdot x=\sum_j w_jx_j$ collapses the $D$ features to one number, the pre-activation $z$ ; the bias is a single scalar added to that number, not to each feature. Then $\sigma$ maps $z$ into $(0,1)$ — a value that can be read as a probability, bounded for every $z$ however extreme. Computing it takes one precaution, in the next hint.

The dot product is a library call, not a loop: w @ x in NumPy, w.dot(x) in Eigen.

Then $\sigma$ on the resulting scalar — but not written literally as 1/(1 + exp(-z)), which overflows at $z=-1000$ because $e^{1000}$ is infinity. Pick the branch whose exponent is never positive:

\sigma(z)=\frac{1}{1+e^{-|z|}}\ \ (z\ge 0), \qquad \sigma(z)=\frac{e^{-|z|}}{1+e^{-|z|}}\ \ (z<0)

Both are algebraically the same function — multiply the first form's numerator and denominator by $e^{z}$ to get the second — and both rest on the one safe quantity $e^{-|z|}$ , which always lands in $(0,1]$ .

Python

def neuron(w: np.ndarray, x: np.ndarray, b: float) -> float:
    """One sigmoid neuron: a = sigmoid(w . x + b).

    w: [D] one weight per input feature
    x: [D] the input features for a single sample
    b: scalar bias, added after the dot product
    Returns the activation a, a scalar float in (0, 1).

    Compute sigma in the overflow-safe form: with e = exp(-|z|), which always
    lands in (0, 1], sigma(z) is 1 / (1 + e) for z >= 0 and e / (1 + e) for
    z < 0. Written literally as 1 / (1 + exp(-z)) it overflows at z = -1000.
    """
    z = float(w @ x) + b
    e = np.exp(-abs(z))  # in (0, 1], never overflows
    return 1.0 / (1.0 + e) if z >= 0.0 else e / (1.0 + e)

A single neuron's forward pass

Dense layer on a batch