Continuous Dense Distributional Word Model Training Algorithm

A Continuous Dense Distributional Word Model Training Algorithm is a dense distributional text-item model training algorithm that can be implemented into a continuous dense distributional word model training system (to solve a continuous dense distributional word model training task).

• AKA: Distributional Word Embedding Algorithm.
• Context:
  • ....
• Example(s):
  • (Levy & Goldberg, 2014)
  • (Mikolov & al, 2013b).
  • (GloVe Algorithm).
• Counter-Example(s):
  • a Discrete Dense Distributional Phrase Model Training Algorithm.
  • a Next-Word Model Training Algorithm.
  • a Language Model Training Algorithm.
  • a Sentence Embedding Algorithm, such as (Le & Mikolov, 2014).
  • a Document Embedding Algorithm, such as (Le & Mikolov, 2014).
• See: Embedding Algorithm, SGNS Algorithm, Word Embeddings.

---

---

References

2014

• (Levy & Goldberg, 2014) ⇒ Omer Levy, and Yoav Goldberg. (2014). "Neural Word Embedding As Implicit Matrix Factorization." In: Advances in Neural Information Processing Systems.
• (Goldberg & Levy, 2014) ⇒ Yoav Goldberg, and Omer Levy. (2014). "word2vec Explained: Deriving Mikolov Et Al.'s Negative-sampling Word-embedding Method." In: arXiv preprint arXiv:1402.3722.
• (Mikolov & al, 2014) => Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S. Corrado, and Jeff Dean. (2014). "Distributed Representations of Words and Phrases and their Compositionality." In: Advances in Neural Information Processing Systems, 26.

2013

• (Chelba & al, 2013) ⇒ Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillipp Koehn, and Tony Robinson. (2013). "One Billion Word Benchmark for Measuring Progress in Statistical Language Modeling." Technical Report, Google Research.
• (Mikolov & al, 2013a) => Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. (2013). "Efficient Estimation of Word Representations in Vector Space." CoRR, abs/1301.3781, 2013.
• (Mikolov & al, 2013b) ⇒ Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig. (2013). "Linguistic Regularities in Continuous Space Word Representations.." In: HLT-NAACL.

2003

• (Bengio & al, 2003a) ⇒ Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Janvin. (2003). "A Neural Probabilistic Language Model." In: The Journal of Machine Learning Research, 3.
  • QUOTE: A goal of statistical language modeling is to learn the joint probability function of sequences of words in a language. This is intrinsically difficult because of the curse of dimensionality: a word sequence on which the model will be tested is likely to be different from all the word sequences seen during training. Traditional but very successful approaches based on n-grams obtain generalization by concatenating very short overlapping sequences seen in the training set. We propose to fight the curse of dimensionality by learning a distributed representation for words which allows each training sentence to inform the model about an exponential number of semantically neighboring sentences.

1986

• (Hinton, 1986) ⇒ Geoffrey E. Hinton. (1986). "Learning Distributed Representations of Concepts." In: Proceedings of the eighth annual conference of the cognitive science society.
  • QUOTE: Concepts can be represented by distributed patterns of activity in networks of neuron-like units. One advantage of this kind of representation is that it leads to automatic generalization.

---