Maximum Entropy-based Learning Algorithm

From GM-RKB
Revision as of 04:07, 29 April 2012 by Gmelli (talk | contribs)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to navigation Jump to search

A Maximum Entropy-based Learning Algorithm is a Supervised Discriminative Classification Algorithm that favors class predictions with Maximum Entropy (least biased estimates).



References

2009

  • http://www.cs.cmu.edu/afs/cs/user/aberger/www/html/tutorial/node2.html
    • The maximum entropy method answers both these questions. Intuitively, the principle is simple: model all that is known and assume nothing about that which is unknown. In other words, given a collection of facts, choose a model which is consistent with all the facts, but otherwise as uniform as possible. This is precisely the approach we took in selecting our model at each step in the above example. ... In its most general formulation, maximum entropy can be used to estimate any probability distribution. In this paper we are interested in classication; thus we limit our further discussion to learning conditional distributions from labeled training data. Specically, we learn the conditional distribution of the class label given a document.
  • http://homepages.inf.ed.ac.uk/s0450736/maxent.html
  • MEGA Optimization Package

2004

1996

1989

1979

1957

  • (Jaynes, 1957) ⇒ E. T. Jaynes. (1957). "Information Theory and Statistical Mechanics.
    • "Information theory provides a constructive criterion for setting up probability distributions on the basis of partial knowledge, and leads to a type of statistical inference which is called the maximum entropy estimate. It is least biased estimate possible on the given information; i.e., it is maximally noncommittal with regard to missing information.