Text-String Probability Function Training Task: Difference between revisions

Revision as of 14:07, 21 March 2018

AKA: Statistical Language Modeling.
Context:
- It can range from being a Heuristic Language Modeling Task to being a Data-Driven Language Modeling Task.
- It can be solved by a Language Modeling System (that applies a language modeling algorithm).
Counter-Example(s):
- Document Modeling.
- Word Vector Space Modeling Task.
See: n-Gram.

(Collins, 2013a) ⇒ Michael Collins. (2013). "Chapter 1 - Language Modeling." Course notes for NLP by Michael Collins, Columbia University.
- QUOTE: Definition 1 (Language Model) A language model consists of a finite set [math]\displaystyle{ \mathcal{V} }[/math], and a function [math]\displaystyle{ p(x_1, x_2, ... x_n) }[/math] such that:
  1. For any [math]\displaystyle{ \lt x_1 ... x_n\gt \in \mathcal{V}^{\dagger}, p(x_1,x_2,... x_n) \ge 0 }[/math]
  2. In addition, [math]\displaystyle{ \Sigma_{\lt x_1 ... x+n\gt } \in \mathcal{V}^{\dagger} p(x1; x2, ... xn) = 1 }[/math]
- Hence [math]\displaystyle{ p(x_1,x_2,... x_n) }[/math] is a probability distribution over the sentences in [math]\displaystyle{ \mathcal{V}^{\dagger} }[/math].

(Goodman, 2001) ⇒ Joshua T. Goodman. (2001). “A Bit of Progress in Language Modeling." In: Computer Speech & Language, 15(4). doi:10.1006/csla.2001.0174
- QUOTE: The goal of a language model is to determine the probability of a word sequence [math]\displaystyle{ w_1...w_n, P (w_1...w_n) }[/math]. This probability is typically broken down into its component probabilities: : [math]\displaystyle{ P (w_1...w_i) = P (w_1) × P (w_2 \mid w_1) ×... × P (w_i \mid w_1...w_{i−1}) }[/math] Since it may be difficult to compute a probability of the form [math]\displaystyle{ P(w_i \mid w_1...w_{i−1}) }[/math] for large i, we typically assume that the probability of a word depends on only the two previous words, the trigram assumption: : [math]\displaystyle{ P (w_i \mid w_1...w_{i−1}) ≈ P (w_i \mid w_i−2w_{i−1}) }[/math] which has been shown to work well in practice.

@@ Line 15: / Line 15: @@
 * ([[Collins, 2013a]]) &rArr; [[Michael Collins]]. (2013). "[http://www.cs.columbia.edu/~mcollins/lm-spring2013.pdf Chapter 1 - Language Modeling]." Course notes for NLP by Michael Collins, Columbia University.
 ** QUOTE: Definition 1 ([[Language Model]]) A [[language model]] consists of a [[finite set]] <math>\mathcal{V}</math>, and a </i>[[vector function|function]]</i> <math>p(x_1, x_2, ... x_n)</math> such that:
-**# For any <math><x_1 ...x_n> \in \mathcal{V}^{\dagger}, p(x_1,x_2,... x_n) \ge 0</math>
+**# For any <math>\lt x_1 ... x_n> \in \mathcal{V}^{\dagger}, p(x_1,x_2,... x_n) \ge 0</math>
-**# In addition, <math>\Sigma_{<x_1 ... x+n>} \in \mathcal{V}^{\dagger} p(x1; x2, ... xn) = 1</math>
+**# In addition, <math>\Sigma_{\lt x_1 ... x+n>} \in \mathcal{V}^{\dagger} p(x1; x2, ... xn) = 1</math>
 ** Hence <math>p(x_1,x_2,... x_n)</math> is a [[probability distribution]] over the [[sentence]]s in <math>\mathcal{V}^{\dagger}</math>.