Exploding Gradient Problem

A Exploding Gradient Problem is a Machine Learning Training Algorithm problem that arises when using gradient descent and backpropagation.

• Example(s):
  • Figure 3 in Grosse (2017).
• Counter-Example(s):
  • Vanishing Gradient Problem.
• See: Recurrent Neural Network, Deep Neural Network Training Algorithm.

---

---

References

2017

• (Grosse, 2017) ⇒ Roger Grosse (2017). "Lecture 15: Exploding and Vanishing Gradients". In: CSC321 Course, University of Toronto.
  • QUOTE: To make this story more concrete, consider the following RNN, which uses the tanh activation function:

Figure 3 shows the function computed at each time step, as well as the function computed by the network as a whole. From this figure, you can see which regions have exploding or vanishing gradients.

Figure 3: (left) The function computed by the RNN at each time step, (right) the function computed by the network.

---