Multiclass Cross-Entropy Measure

From GM-RKB
(Redirected from relative or cross entropy)
Jump to: navigation, search

A Multiclass Cross-Entropy Measure is a dispersion measure which measures the average number of bits needed to identify an event from a set of possibilities.



References

2017

  • (Wikipedia, 2017) ⇒ https://en.wikipedia.org/wiki/cross_entropy Retrieved:2017-6-7.
    • In information theory, the cross entropy between two probability distributions [math] p [/math] and [math] q [/math] over the same underlying set of events measures the average number of bits needed to identify an event drawn from the set, if a coding scheme is used that is optimized for an "unnatural" probability distribution [math] q [/math] , rather than the "true" distribution [math] p [/math] .

      The cross entropy for the distributions [math] p [/math] and [math] q [/math] over a given set is defined as follows: : [math] H(p, q) = \operatorname{E}_p[-\log q] = H(p) + D_{\mathrm{KL}}(p \| q),\! [/math] where [math] H(p) [/math] is the entropy of [math] p [/math] , and [math] D_{\mathrm{KL}}(p \| q) [/math] is the Kullback–Leibler divergence of [math] q [/math] from [math] p [/math] (also known as the relative entropy of p with respect to q — note the reversal of emphasis).

      For discrete [math] p [/math] and [math] q [/math] this means : [math] H(p, q) = -\sum_x p(x)\, \log q(x). \! [/math] The situation for continuous distributions is analogous. We have to assume that [math] p [/math] and [math] q [/math] are absolutely continuous with respect to some reference measure [math] r [/math] (usually [math] r [/math] is a Lebesgue measure on a Borel σ-algebra). Let [math] P [/math] and [math] Q [/math] be probability density functions of [math] p [/math] and [math] q [/math] with respect to [math] r [/math] . Then : [math] -\int_X P(x)\, \log Q(x)\, dr(x) = \operatorname{E}_p[-\log Q]. \! [/math] NB: The notation [math] H(p,q) [/math] is also used for a different concept, the joint entropy of [math] p [/math] and [math] q [/math]

2017

2011a

2011b

2004