SPPMI-SVD Word Model Training Algorithm

A SPPMI-SVD Word Model Training Algorithm is a factorization-based continuous dense distributional word model training algorithm that applies singular value decomposition to a shifted PPMI matrix.

• • …
• Counter-Example(s):
  • GloVe Algorithm.
  • CBOW NNLM, and CBOW NNLM with Negative Sampling.
  • Skip-Gram NNLM, and Skip-Gram NNLM with Negative Sampling.
• See: Weighted Matrix Factorization.

---

---

References

2014

• (Řehůřek, 2014a) ⇒ Radim Řehůřek. (2014-12-23). http://radimrehurek.com/2014/12/making-sense-of-word2vec/
  • QUOTE: ... we can directly take rows of this SPPMI matrix to be the word vectors.

    ... The SPPMI-SVD method simply factorizes the sparse SPPMI matrix using Singular Value Decomposition (SVD), rather than the gradient descent methods of word2vec/GloVe, and uses the (dense) left singular vectors as the final word embeddings.

• (Levy & Goldberg, 2014) ⇒ Omer Levy, and Yoav Goldberg. (2014). “Neural Word Embedding As Implicit Matrix Factorization.” In: Advances in Neural Information Processing Systems (NIPS 2014).
  • QUOTE: ...

---