The steepest descent method, also known as the gradient method, is one of the most fundamental algorithms in unconstrained optimization. It iteratively moves in the direction of the negative gradient, which locally points toward the steepest decrease of the objective function. A line search determines the optimal step size at each iteration. While simple and computationally cheap, the method often exhibits a characteristic zigzag path, especially near the minimum of ill-conditioned problems, leading to slow convergence. This article explains the underlying theory, illustrates the geometric behavior with a convex quadratic function, and provides a complete C++ implementation. Understanding steepest descent is essential for grasping more advanced optimization techniques like conjugate gradient and quasi-Newton methods. It remains a valuable baseline for comparing and debugging more sophisticated algorithms in machine learning and numerical computing.
A clear walkthrough of the steepest descent method for unconstrained optimization, covering its zigzag convergence behavior and a practical C++ example.