View source for Backpropagation of Errors (BP)-based Training Algorithm

A [[Backpropagation of Errors (BP)-based Training Algorithm]] is a [[feed-forward neural network training algorithm|feed-forward]] [[supervised neural network training algorithm]] that applies a [[delta rule]] to feed the intermediate [[NNet]]s [[loss function]] [[gradient]] (which is used to update the [[neuron weight]]s to [[minimize the loss function]]).
* <B>Context:</B>
** It can (typically) be guaranteed to convergence to a solution (although this may be a local minimum on the [[error surface]]).
** It can (typically) be used with [[Differentiable Activation Function]]s like [[ReLU]], [[Sigmoid]], and [[Tanh]] (to ensure the gradients can be propagated through the network).
** It can (often) be a [[Slow Algorithm]] (because it can converge slowly to a solution).
** It can (often) use [[Normalization Technique]]s such as [[Batch Normalization]] (to improve training stability and speed).
** It can (often) use [[Regularization Technique]]s like [[dropout technique]] (to prevent overfitting in large networks).
** ...
** It can range from being a [[Batch Backpropagation Algorithm]] to being an [[Iterative Backpropagation Algorithm]].
** ...
** It can make use of the [[Chain Rule]] to compute gradients for each layer in the network.
** It can use [[Gradient Clipping Technique]]s to mitigate exploding gradients.
** It can benefit from using [[Gradient Descent Momentum]] in order to help escape local minima on the error surface.
** It can face issues with [[saddle points]], where gradients are near zero, slowing down convergence significantly.
** It can be enhanced by using [[Adaptive Learning Rate Algorithms]] like [[Adam]] or [[RMSprop]] to dynamically adjust learning rates during training.
** ...
* <B>Example(s):</B>
** [[Stochastic Gradient Descent Backpropagation Algorithm]].
** [[Back-Propagation Through Time (BPTT) Algorithm]].
** [[BRNN Backpropagation Algorithm]].
** as applied to [[Convolutional Neural Networks (CNNs)]] trained for image classification using backpropagation, which iteratively adjusts filters to minimize the classification error.
** as applied to [[Recurrent Neural Networks (RNNs)]] that apply [[Back-Propagation Through Time (BPTT)]] for sequence prediction tasks, adjusting weights based on errors propagated through time steps.
** ...
* <B>Counter-Example(s):</B>
** [[Perceptron Training Algorithm]].
** [[Human Brain Learning Algorithm]].
** [[Boltzmann Machine]].
** [[Evolutionary Algorithm]]s, which do not rely on gradient-based optimization but rather on genetic operations like mutation and crossover.
** [[Reinforcement Learning Algorithm]]s, where learning is driven by reward signals rather than error backpropagation.
** [[Support Vector Machines (SVMs)]], which use a different optimization approach based on maximizing the margin between classes rather than minimizing a loss function via backpropagation.
* <B>See:</B> [[Gradient Descent Algorithm]], [[Chain Rule]], [[Activation Function]], [[Gradient Vanishing Problem]], [[Learning Rate Scheduling]], [[Second-Order Optimization Methods]]

----
----