Neural Network with Attention Mechanism

From GM-RKB
Jump to navigation Jump to search

A Neural Network with Attention Mechanism is Memory-based Neural Network that includes an attention mechanism.



References

2018a

2018b

2017

2017a

2017b

  • (Synced Review, 2017) ⇒ Synced (2017). “A Brief Overview of Attention Mechanism." In: Medium - Synced Review Blog Post.
    • QUOTE: ... Attention is simply a vector, often the outputs of dense layer using softmax function. Before Attention mechanism, translation relies on reading a complete sentence and compress all information into a fixed-length vector, as you can imagine, a sentence with hundreds of words represented by several words will surely lead to information loss, inadequate translation, etc. However, attention partially fixes this problem. It allows machine translator to look over all the information the original sentence holds, then generate the proper word according to current word it works on and the context. It can even allow translator to zoom in or out (focus on local or global features).

2016a

2016 HierarchicalAttentionNetworksfo Fig2.png
Figure 2: Hierarchical Attention Network.

2016c

2016 BidirectionalRecurrentNeuralNet Fig1.png
Figure 1: Description of the model predicting punctuation [math]\displaystyle{ y_t }[/math] at time step [math]\displaystyle{ t }[/math] for the slot before the current input word $x_t$.

2015a

2015b

2015c

2015d