Hierarchical Recurrent Neural Network

A Hierarchical Recurrent Neural Network is a Hierarchical Attention Network that is based on a recurrent neural network architecture.

• Example(s):
  • a Document Classification Hierarchical Attention Network (e.g. Yang et al., 2016),
  • …
• Counter-Example(s):
  • a Hierarchical Bayesian Network,
  • a Pointer Network (Ptr-Net),
  • a Pointer-Generator Network,
  • a Sequence-to-Sequence Neural Network with Attention Mechanism,
  • a Transformer Neural Network.
• See: Long Short-Term Memory, Recurrent Neural Network, Convolutional Neural Network, Gating Mechanism.

---

---

References

2016

• (Yang et al., 2016) ⇒ Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. (2016). “Hierarchical Attention Networks for Document Classification.” In: Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-2016).
  • QUOTE: We propose a hierarchical attention network for document classification. Our model has two distinctive characteristics: (i) it has a hierarchical structure that mirrors the hierarchical structure of documents; (ii) it has two levels of attention mechanisms applied at the word and sentence level, enabling it to attend differentially to more and less important content when constructing the document representation(...)

    The overall architecture of the Hierarchical Attention Network (HAN) is shown in Fig. 2. It consists of several parts: a word sequence encoder, a word-level attention layer, a sentence encoder and a sentence-level attention layer. We describe the details of different components in the following sections.

Figure 2: Hierarchical Attention Network.

---