Gradient Descent-based Learning Algorithm

A Gradient Descent-based Learning Algorithm is a supervised learning algorithm that is a gradient-descent optimization algorithm.

• Context:
  • …
• Example(s):
  • an Accelerated Gradient Descent (AGD),
  • an Adaptive Gradient Algorithm (AdaGrad),
  • an Adaptive Learning Rate Algorithm (ADADELTA),
  • an Adaptive Moment Estimation Algorithm (Adam),
  • a Gradient-Descent Boosted Learning Algorithm.
  • a Mini-Batch Gradient Descent Algorithm (MBGD).
  • a Momentum Gradient Descent (MGD),
  • a Root Mean Square Propagation Algorithm (RMSprop);
  • a Stochastic Gradient Descent Algorithm.
  • …
• Counter-Example(s):
  • Convex Optimization-based Learning Algorithm.
• See: Perceptron Algorithm, Gradient Descent-based Learning System.

---

---

References

2018

• (Wijaya et al., 2018) ⇒ Galih Praja Wijaya, Dendi Handian, Imam Fachmi Nasrulloh, Lala Septem Riza, Rani Megasari, Enjun Junaeti (2018), "gradDescent: Gradient Descent for Regression Tasks", "Reference manual (PDF).
  • QUOTE: An implementation of various learning algorithms based on Gradient Descent for dealing with regression tasks. The variants of gradient descent algorithm are: Mini-Batch Gradient Descent (MBGD), which is an optimization to use training data partially to reduce the computation load. Stochastic Gradient Descent (SGD), which is an optimization to use a random data in learning to reduce the computation load drastically. Stochastic Average Gradient (SAG), which is a SGD-based algorithm to minimize stochastic step to average. Momentum Gradient Descent (MGD), which is an optimization to speed-up gradient descent learning. Accelerated Gradient Descent (AGD), which is an optimization to accelerate gradient descent learning. Adagrad, which is a gradient-descent-based algorithm that accumulate previous cost to do adaptive learning. Adadelta, which is a gradient-descent-based algorithm that use hessian approximation to do adaptive learning. RMSprop, which is a gradient-descent-based algorithm that combine Adagrad and Adadelta adaptive learning ability. Adam, which is a gradient-descent-based algorithm that mean and variance moment to do adaptive learning. Stochastic Variance Reduce Gradient (SVRG), which is an optimization SGD-based algorithm to accelerates the process toward converging by reducing the gradient. Semi Stochastic Gradient Descent (SSGD),which is a SGD-based algorithm that combine GD and SGD to accelerates the process toward converging by choosing one of the gradients at a time. Stochastic Recursive Gradient Algorithm (SARAH), which is an optimization algorithm similarly SVRG to accelerates the process toward converging by accumulated stochastic information. Stochastic Recursive Gradient Algorithm+ (SARAHPlus), which is a SARAH practical variant algorithm to accelerates the process toward converging provides a possibility of earlier termination.

1998

• (LeCun et al., 1998) ⇒ Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. (1998). “Gradient-based Learning Applied to Document Recognition." doi:10.1109/5.726791

1990

• (Williams et al., 1990) ⇒ Ronald J. Williams, and Jing Peng. (1990). “An Efficient Gradient-based Algorithm for on-line Training of Recurrent Network Trajectories." Neural computation 2, no. 4

---