# Gradient Descent-based Learning Algorithm

(Redirected from gradient descent learning)

A Gradient Descent-based Learning Algorithm is a supervised learning algorithm that is a gradient-descent optimization algorithm.

**Context:**- …

**Example(s):**- an Accelerated Gradient Descent (AGD),
- an Adaptive Gradient Algorithm (AdaGrad),
- an Adaptive Learning Rate Algorithm (ADADELTA),
- an Adaptive Moment Estimation Algorithm (Adam),
- a Gradient-Descent Boosted Learning Algorithm.
- a Mini-Batch Gradient Descent Algorithm (MBGD)
- a Momentum Gradient Descent (MGD),
- a Root Mean Square Propagation Algorithm (RMSprop);
- a Stochastic Gradient Descent Algorithm.

**Counter-Example(s):****See:**Perceptron Algorithm, Gradient Descent-based Learning System.

## References

### 2018

- (Wijaya et al., 2018) ⇒ Galih Praja Wijaya, Dendi Handian, Imam Fachmi Nasrulloh, Lala Septem Riza, Rani Megasari, Enjun Junaeti (2018), "gradDescent: Gradient Descent for Regression Tasks", "Reference manual (PDF).
- QUOTE: An implementation of various learning algorithms based on Gradient Descent for dealing with regression tasks. The variants of gradient descent algorithm are: Mini-Batch Gradient Descent (MBGD), which is an optimization to use training data partially to reduce the computation load. Stochastic Gradient Descent (SGD), which is an optimization to use a random data in learning to reduce the computation load drastically. Stochastic Average Gradient (SAG), which is a SGD-based algorithm to minimize stochastic step to average. Momentum Gradient Descent (MGD), which is an optimization to speed-up gradient descent learning. Accelerated Gradient Descent (AGD), which is an optimization to accelerate gradient descent learning. Adagrad, which is a gradient-descent-based algorithm that accumulate previous cost to do adaptive learning. Adadelta, which is a gradient-descent-based algorithm that use hessian approximation to do adaptive learning. RMSprop, which is a gradient-descent-based algorithm that combine Adagrad and Adadelta adaptive learning ability. Adam, which is a gradient-descent-based algorithm that mean and variance moment to do adaptive learning. Stochastic Variance Reduce Gradient (SVRG), which is an optimization SGD-based algorithm to accelerates the process toward converging by reducing the gradient. Semi Stochastic Gradient Descent (SSGD),which is a SGD-based algorithm that combine GD and SGD to accelerates the process toward converging by choosing one of the gradients at a time. Stochastic Recursive Gradient Algorithm (SARAH), which is an optimization algorithm similarly SVRG to accelerates the process toward converging by accumulated stochastic information. Stochastic Recursive Gradient Algorithm+ (SARAHPlus), which is a SARAH practical variant algorithm to accelerates the process toward converging provides a possibility of earlier termination.

### 1998

- (LeCun et al., 1998) ⇒ Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. (1998). “Gradient-based Learning Applied to Document Recognition." doi:10.1109/5.726791

### 1990

- (Williams et al., 1990) ⇒ Ronald J. Williams, and Jing Peng. (1990). “An Efficient Gradient-based Algorithm for on-line Training of Recurrent Network Trajectories." Neural computation 2, no. 4