Neural Network-based Language Model (NLM)

A Neural Network-based Language Model (NLM) is a language model that is neural network model.

• Context:
  • It can be produced by a Neural Language Modeling System (that can solve a neural LM training task).
  • It can range form (typically) being a Pretrained Neural Language Model (LM) to being an Untretrained Neural Language Model (LM).
  • It can range from being a Character-Level Neural Network-based LM to being a Word/Token-Level Neural Network-based LM.
  • It can range from being a Forward Neural Network-based Language Model to being a Backward Neural Network-based Language Model to being a Bi-Directional Neural Network-based Language Model.
  • It can range from (typically) being a Deep NNet-based LM (such as a large NLM) to being a Shallow NNet-based LM.
  • It can range from being a Uni-Lingual NLM to being a Multi-Lingual NLM.
  • …
• Example(s):
  • a Bigram Neural Language Model (previous word is used to predict the current word).
  • an RNN-based Language Model such as:
    • Recurrent Neural Network Character-Level Language Model (Karpathy, 2015),
    • Bengio Neural Language Model (Bengio et al., 2003),
    • rnnlm System,
    • AWD LSTM Language Model,
    • an Universal Language Model Fine-tuning for Text Classification (ULMFiT),
  • an Transformer-based Language Model such as: GPT-2, BERT-based model, ELMo, and Turing-NLG.
  • an Universal Language Model Fine-tuning for Text Classification (ULMFiT).
  • …
• Counter-Example(s):
  • a Text-Substring Probability Function,
  • an N-Gram Language Model,
  • an Exponential Language Model,
  • a Cache Language Model (Jelinek et al., 1991),
  • a Bag-Of-Concepts Model (Cambria & Hussain, 2012),
  • a Positional Language Model (Lv & Zhai, 2009),
  • a Structured Language Model (Chelba and Jelinek, 2000),
  • a Random Forest Language Model (Xu, 2005),
  • a Bayesian Language Model (Teh, 2006),
  • a Class-based Language Model (Brown et al., 1992),
  • a Maximum Likelihood-based Language Model (Goldberg, 2015),
  • a Query Likelihood Model,
  • a Factored Language Model.
• See: Language Modeling Task, Language Modeling System, Natural Language Representation Dataset, Language Modeling Benchmark, Artificial Neural Network, Natural Language Processing Task, Natural Language Understanding Task, Natural Language Inference Task.

---

---

References

2017

• (Daniluk et al., 2017) ⇒ Michał Daniluk, Tim Rocktaschel, Johannes Welbl, and Sebastian Riedel. (2017). “Frustratingly Short Attention Spans in Neural Language Modeling.” In: Proceedings of ICLR 2017.
  • QUOTE: Neural language models predict the next token using a latent representation of the immediate token history. Recently, various methods for augmenting neural language models with an attention mechanism over a differentiable memory have been proposed. For predicting the next token, these models query information from a memory of the recent history which can facilitate learning mid - and long-range dependencies. However, conventional attention mechanisms used in memory-augmented neural language models produce a single output vector per time step.

2015

• (Karpathy, 2015) ⇒ Andrej Karpathy. (2015). “The Unreasonable Effectiveness of Recurrent Neural Networks.” In: Proceedings of Blog post 2015-05-21.
  • QUOTE: ... By the way, together with this post I am also releasing code on Github that allows you to train character-level language models based on multi-layer LSTMs. You give it a large chunk of text and it will learn to generate text like it one character at a time. ...

2003

• (Bengio et al., 2003a) ⇒ Yoshua Bengio, R. Ducharme, Vincent, P., and C. Jauvin. (2003). “A Neural Probabilistic Language Model.” In: Journal of Machine Learning Research, 3(6).
  • QUOTE: A goal of statistical language modeling is to learn the joint probability function of sequences of words in a language. This is intrinsically difficult because of the curse of dimensionality: a word sequence on which the model will be tested is likely to be different from all the word sequences seen during training. … We report on experiments using neural networks for the probability function, showing on two text corpora that the proposed approach significantly improves on state-of-the-art n-gram models, and that the proposed approach allows to take advantage of longer contexts.

---