Gradient descent

Gradient descent is how models learn: step downhill on the cost curve until you reach the bottom.

The gradient points uphill, so you step the opposite way. The learning rate sets step size: too big overshoots, too small crawls. One step: w = w − learning_rate · gradient.