What Is Gradient Descent?
Gradient descent is the workhorse optimization algorithm: to find a minimum of a function, repeatedly step in the direction of steepest decrease, which is the negative gradient. The update rule is
(x, y) ← (x, y) − η ∇f(x, y)
where η (the learning rate or step size) controls how far each step moves. Because the gradient points in the direction of steepest increase, its negative points steepest downhill, so each step lowers the function value, and repeating drives the point toward a local minimum. The whole method rests on one calculus fact: the gradient is the direction of fastest change. This calculator runs the iteration on a function of two variables, showing each step's gradient and new position, tracking the function value downhill, and plotting the descent path. Crucially, it lets you see how the learning rate makes or breaks the method: too small and progress crawls, too large and the iterates overshoot and diverge.
How to Use the Gradient Descent Calculator
Enter the objective f(x, y), a starting point, a learning rate η, and a number of steps. The steps compute the gradient symbolically, then list each iteration's gradient, new position, and function value, followed by a diagnosis of the outcome (converged, still descending, or diverged) with advice on the learning rate. The graph traces the descent path from start to finish. This applies the gradient and the directional derivative, the descent path crosses level curves perpendicularly, and the curvature that governs the best learning rate comes from the Hessian.
Worked Example
Minimize f(x, y) = x² + 3y² from the start (2, 2) with η = 0.1. The gradient is ∇f = (2x, 6y), so at (2, 2) it is (4, 12). The first step is
(2, 2) − 0.1·(4, 12) = (1.6, 0.8)
and the function value drops from 16 to 4.48. Continuing, the iterates spiral in toward the minimum at the origin, where the gradient vanishes. Notice the path is not straight: because the bowl is steeper in the y-direction (coefficient 3) than the x-direction, the y-coordinate shrinks faster, so the path curves. Now try the same problem with η = 0.4 (the second chip): the steps in y overshoot the bottom and bounce to the other side higher than before, and the iterates diverge to infinity. This is the central lesson of gradient descent, the learning rate must be small enough for the function's curvature, and choosing it is the practical art of the method.
Why the Learning Rate Is Everything
Gradient descent's convergence hinges entirely on the learning rate matching the function's curvature. For a quadratic bowl, there is a critical rate above which steps overshoot the minimum and grow: if η exceeds 2 divided by the largest curvature (the largest eigenvalue of the Hessian), the iteration diverges, as the η = 0.4 example shows. Below that threshold, descent converges, but the rate of convergence depends on the ratio of largest to smallest curvature, the condition number. When a bowl is much steeper in one direction than another (an ill-conditioned valley, like x² + 100y²), you are forced to use a tiny learning rate to avoid overshooting the steep direction, which makes progress along the shallow direction agonizingly slow, producing the characteristic zigzag down a narrow ravine. This tension, small steps for stability versus large steps for speed, motivates the entire zoo of advanced optimizers: momentum methods that build up velocity along consistent directions, adaptive methods (AdaGrad, RMSprop, Adam) that scale the step per-coordinate, and second-order methods that use the Hessian to correct the direction. But all of them are refinements of the same core idea computed here: follow the negative gradient downhill. Gradient descent also finds only local minima, it slides into whatever basin it starts in, and it can stall at saddle points where the gradient vanishes without a minimum, both fundamental limitations that shape how the method is deployed in practice.
Common Mistakes to Avoid
- Choosing too large a learning rate. The most common failure: steps overshoot the minimum and the iterates diverge to infinity. If the function value increases instead of decreasing, cut η (often by a factor of ten) and restart.
- Choosing too small a learning rate. Descent still works but crawls, needing thousands of steps for what a well-chosen rate does in dozens. There is a sweet spot tied to the function's curvature.
- Expecting a global minimum. Gradient descent finds a local minimum in the basin of the starting point. For functions with multiple minima, the result depends on where you start; global optimization needs additional strategies.
- Ignoring saddle points. The gradient vanishes at saddles too, so descent can slow dramatically or stall near one, mistaking it for a minimum. Checking the Hessian distinguishes the two.
- Forgetting the negative sign. Descent subtracts the gradient (moves against it). Adding the gradient performs steepest ascent, climbing to a maximum instead, a sign error that sends the iteration exactly the wrong way.
Real-World Applications
Gradient descent is arguably the most important algorithm in modern computing, because it is the engine that trains machine learning models. Every neural network, from image classifiers to large language models, is trained by gradient descent (in the stochastic and mini-batch variants) on a loss function of millions or billions of parameters: backpropagation computes the gradient, and the parameters step downhill to reduce prediction error. The learning-rate tuning demonstrated here on a two-variable bowl is exactly the challenge practitioners face at massive scale, and the adaptive optimizers (Adam and its relatives) that dominate deep learning are direct descendants of the basic method, engineered to handle the ill-conditioning and stochasticity of real loss landscapes.
Frequently Asked Questions
How does gradient descent work?
It repeatedly steps in the direction of the negative gradient, the steepest downhill direction, scaled by a learning rate: (x, y) ← (x, y) − η∇f. Each step lowers the function value, and iterating drives the point toward a local minimum where the gradient vanishes.
What is the learning rate?
The step-size parameter η that controls how far each iteration moves along the negative gradient. It is the most important tuning choice: too large causes overshooting and divergence, too small causes painfully slow progress. The ideal value depends on the function's curvature.
Why does a large learning rate cause divergence?
Because a large step overshoots the minimum and lands on the far side higher than it started, and the next step overshoots even more, so the iterates grow without bound. For a quadratic, divergence occurs when η exceeds 2 divided by the largest curvature (Hessian eigenvalue).
Does gradient descent find the global minimum?
No, only a local minimum, the bottom of whatever basin the starting point lies in. For functions with several minima, different starts reach different results. Finding the global minimum requires extra techniques like random restarts, simulated annealing, or convexity (which makes local and global coincide).
Why does the descent path curve instead of going straight to the minimum?
Because the gradient points steepest-downhill locally, not directly at the minimum unless the bowl is perfectly circular. When the function is steeper in one direction, that coordinate shrinks faster, bending the path. Elongated (ill-conditioned) bowls produce pronounced zigzagging.
What is an ill-conditioned problem?
One where the function is far steeper in some directions than others, giving a large condition number (ratio of largest to smallest curvature). Gradient descent must use a small learning rate to stay stable in the steep direction, making progress in the shallow direction very slow, hence the zigzag down a narrow valley.
How is gradient descent used in machine learning?
It trains models by minimizing a loss function over the parameters. Backpropagation computes the loss gradient, and the parameters step downhill to reduce error. Stochastic and mini-batch variants use gradient estimates from data samples, and adaptive optimizers like Adam refine the basic update for large-scale training.
What happens at a saddle point?
The gradient is zero there, so descent slows or stalls even though the point is not a minimum (the function rises in some directions and falls in others). Descent can eventually escape along a downhill direction, but saddles can trap or slow the method, a known challenge in high-dimensional optimization.
How do momentum and adaptive methods improve on plain descent?
Momentum accumulates velocity along consistent gradient directions, smoothing the zigzag and speeding progress through valleys. Adaptive methods (AdaGrad, RMSprop, Adam) scale the step per coordinate based on past gradients, effectively using a different learning rate for each direction to handle ill-conditioning automatically.
How do I choose a good learning rate?
Start moderate and watch the function value: if it decreases steadily, the rate is workable; if it increases or oscillates, reduce it. A common practice is to try values spaced by factors of ten. For quadratics, the safe range is below 2 over the largest curvature; learning-rate schedules that shrink η over time also help.