Dynamic Neural Turing Machine (D-NTM)

A Dynamic Neural Turing Machine (D-NTM) is a Neural Turing Machine that is based on a Least Recently Used (LRU) Addressing Mechanism.

• Context:
  • It includes a recurrent neural network controller and external memory module.
  • It can range from being a Discrete D-NTM to being a Continuous D-NTM.
• Example(s):
  • …
• Counter-Example(s):
  • a Neural Machine Translation (NMT) Network,
  • a Hierarchical Attention Network,
  • a Gated Convolutional Neural Network with Segment-level Attention Mechanism (SAM-GCNN),
  • a Convolutional Neural Network with Segment-level Attention Mechanism (SAM-CNN),
  • a Bidirectional Recurrent Neural Network with Attention Mechanism,
  • a Sparse Access Memory Neural Network (SAM-ANN).
• See: Dynamic Memory Network, Turing Machine, Programmable Computer, Auxiliary Memory, Gradient Descent, Long Short-Term Memory, Differentiable Neural Computer, Attention Mechanism.

---

---

References

2018

• (Gulcehre et al., 2018) ⇒ Caglar Gulcehre, Sarath Chandar, Kyunghyun Cho, and Yoshua Bengio. (2018). "Dynamic Neural Turing Machine with Continuous and Discrete Addressing Schemes". In: Neural Computation Journal, 30(4). ISBN:0899-7667 doi:10.1162/neco_a_01060
  • QUOTE: We propose a variation of an NTM, a dynamic neural Turing machine (D-NTM), which employs learnable and location-based addressing.

    (...) The proposed dynamic neural Turing machine (D-NTM) extends the neural Turing machine (NTM; Graves et al., 2014). The D-NTM consists of two main modules: a controller and a memory. The controller, which is often implemented as a recurrent neural network, issues a command to the memory so as to read, write to, and erase a subset of memory cells.

    

    Figure 1: A graphical illustration of the proposed dynamic neural Turing machine with the recurrent-controller. The controller receives the fact as a continuous vector encoded by a recurrent neural network, computes the read and write weights for addressing the memory. If the D-NTM automatically detects that a query has been received, it returns an answer and terminates.

---