Activations & Losses

The logistic sigmoid squashes any real number into $(0, 1)$ , applied elementwise over an array:

\sigma(z) = \frac{1}{1 + e^{-z}}

Computed literally, $e^{-z}$ overflows for large negative $z$ (try $z = -1000$ ). For $z < 0$ use the algebraically equal form $\sigma(z) = \frac{e^{z}}{1 + e^{z}}$ , so the exponent is never positive.

`z`	result
`[0]`	`[0.5]`
`[-2, 2]`	`[0.1192, 0.8808]`
`[[-1000, 1000]]`	`[[0, 1]]` (no overflow)

The two forms are the same number: multiply numerator and denominator of the first by $e^{z}$ . One branch for $z \ge 0$ , one for $z < 0$ , and the output keeps the shape of $z$ . Two identities the tests check: $\sigma(0) = 0.5$ and $\sigma(-z) = 1 - \sigma(z)$ .

Both branches come from one safe quantity $e^{-|z|} \in (0, 1]$ : $\sigma(z) = 1/(1+e^{-|z|})$ for $z \ge 0$ and $e^{-|z|}/(1+e^{-|z|})$ for $z < 0$ . In NumPy, np.exp(-np.abs(z)) then np.where(z >= 0, a, b) picks a or b per element; it evaluates both, which is fine here because neither overflows. In Eigen, Z.unaryExpr([](double z) { ... }) applies a lambda to every coefficient and yields a matrix of the same shape.

Python

def sigmoid(z: np.ndarray) -> np.ndarray:
    """Logistic sigmoid, elementwise and numerically stable.

    z: any shape, e.g. [B, H]
    Returns an array of the same shape with sigma(z) = 1 / (1 + e^{-z}) in (0, 1).
    For z < 0 use the equal form e^{z} / (1 + e^{z}) so nothing overflows,
    even at z = -1000.
    """
    e = np.exp(-np.abs(z))  # in (0, 1], never overflows
    return np.where(z >= 0, 1.0 / (1.0 + e), e / (1.0 + e))

Sigmoid

ReLU and its derivative