Bahdanau-Cho-Bengio Neural Machine Translation System

A Bahdanau-Cho-Bengio Neural Machine Translation System is a Neural Machine Translation System that can solve a Bahdanau-Cho-Bengio Neural Machine Translation Task by training a RNNsearch to align and translate text simultaneously.

• AKA: Bahdanau's Neural Machine Translation System.
• Context:
  • It was developed by Bahdanau et al. (2015).
  • System's Architecture:
  • It is based on RNNsearch that consists of an encoder-decoder neural network with a alignment mechanism containing:
    • 1000 RNN gated hidden units each ;
    • 620 word embeddings;
    • 1000 alignment model hidden units;
    • 500 Deep output maxout hidden layers;
  • Training and other ML Tools :
    • Parameter initialization setting:
      • alignment + bias vectors initial value is set to 0;
      • RNN weight matrices are initialized by sampling from a Gaussian distribution $\mathcal{N}(0, 0.001^2)$, other weight matrices are initiated by sampling from a Gaussian distribution $\mathcal{N}(0,0.01^2)$
    • It uses a beam search to find a translation that approximately maximizes the conditional probability;
    • It uses a minibatch stochastic gradient descent (SGD) algorithm together with Adadelta (Zeiler, 2012) to train each NMT model.
    • It uses MOSES (Koehn et al., 2007) tokenization script.
  • …
• Example(s):
  • …
• Counter-Example(s):
  • Lee-Krahmer-Wubben Data-To-Text Generation System,
  • Clark-Ji-SmithNeural Narrative Text Generation System,
  • LSPGS Wikipedia Long Sentences Summarization System,
  • MaskGAN System,
  • SentencePiece System.
• See: Neural Machine Translation System, Neural Text Generation System, Natural Language Processing System, Neural Encoder-Decoder System.

---

---

References

2015

• (Bahdanau et al., 2015) ⇒ Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. (2015). “Neural Machine Translation by Jointly Learning to Align and Translate.” In: Proceedings of the Third International Conference on Learning Representations, (ICLR-2015).

2012

• (Zeiler, 2012) ⇒ Matthew D. Zeiler. (2012). “ADADELTA: An Adaptive Learning Rate Method.” In: e-print arXiv:1212.5701.

2007

• (Koehn et al., 2007) ⇒ Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondrej Bojar, Alexandra Constantin, and Evan Herbst. (2007). "Moses: Open Source Toolkit for Statistical Machine Translation". In: Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics Companion Volume Proceedings of the Demo and Poster Sessions (ACL 2007).

---