Imagine walking downhill in fog, one step at a time.
Direct answer
Gradient descent is how many models learn. It measures how wrong the model is (the loss), calculates which direction reduces it fastest (the gradient) and moves the parameters a small step that way. The learning rate sets the step size. Repeat until the loss stops improving.
A tiny example
# minimise f(x) = (x - 3) ** 2
x = 0.0
lr = 0.1
for step in range(50):
gradient = 2 * (x - 3)
x = x - lr * gradient
print(round(x, 3)) # close to 3
Learning rate matters
| Too small | Too large |
|---|---|
| very slow | overshoots or diverges |
Variants
Stochastic and mini-batch gradient descent use subsets of data per step, which is faster for large datasets.
Where Teyro fits
Teyro's Machine Learning for Beginners course turns this into short lessons you can play in a few minutes a day. It has 200 lessons, and you can start free.
The bottom line
Take small steps downhill. That is how most models learn.
