Gradient Descent

Gradient descent moves every parameter a little way against its gradient: $\theta \leftarrow \theta - \eta\,g$ , where $g = \nabla_\theta L$ is the gradient of the loss and $\eta$ (the learning rate) scales the step. Return the updated parameters as a new array; the input is left untouched.

`params`	`grads`	`lr`	result
`[1, -2]`	`[0.5, 0.5]`	`0.1`	`[0.95, -2.05]`
`[3]`	`[6]`	`0.25`	`[1.5]`
`[0.5, 0.5]`	`[0, 0]`	`1`	`[0.5, 0.5]`

Entry by entry, $\theta_k \leftarrow \theta_k - \eta\,g_k$ : each coordinate moves against its own gradient entry, all scaled by the same $\eta$ . The gradient points where the loss rises fastest, so subtracting a small multiple of it lowers the loss — on $f(\theta) = \theta^2$ with $\theta = 3$ , $g = 6$ and $\eta = 0.1$ the step lands on $2.4$ , where $f$ is smaller.

Whole-array arithmetic does the loop and allocates the result: NumPy params - lr * grads builds a new array (an in-place params -= lr * grads would mutate the caller's array); Eigen params - lr * grads assigned to a fresh Eigen::VectorXd does the same.

Python

def gradient_step(params: np.ndarray, grads: np.ndarray, lr: float) -> np.ndarray:
    """One gradient-descent update: theta <- theta - lr * g.

    params: [P] current parameter values
    grads:  [P] gradient of the loss with respect to each parameter
    lr:     learning rate (step size)
    Returns the updated parameters as a NEW [P] array; `params` must not be modified.
    """
    return params - lr * grads

One gradient step

Read a learning-rate trace