# Multiclass Cross-Entropy Measure

(Redirected from cross-entropy function)

A Multiclass Cross-Entropy Measure is a dispersion measure which measures the average number of bits needed to identify an event from a set of possibilities.

## References

### 2017

• (Wikipedia, 2017) ⇒ https://en.wikipedia.org/wiki/cross_entropy Retrieved:2017-6-7.
• In information theory, the cross entropy between two probability distributions $\displaystyle{ p }$ and $\displaystyle{ q }$ over the same underlying set of events measures the average number of bits needed to identify an event drawn from the set, if a coding scheme is used that is optimized for an "unnatural" probability distribution $\displaystyle{ q }$ , rather than the "true" distribution $\displaystyle{ p }$ .

The cross entropy for the distributions $\displaystyle{ p }$ and $\displaystyle{ q }$ over a given set is defined as follows: : $\displaystyle{ H(p, q) = \operatorname{E}_p[-\log q] = H(p) + D_{\mathrm{KL}}(p \| q),\! }$ where $\displaystyle{ H(p) }$ is the entropy of $\displaystyle{ p }$ , and $\displaystyle{ D_{\mathrm{KL}}(p \| q) }$ is the Kullback–Leibler divergence of $\displaystyle{ q }$ from $\displaystyle{ p }$ (also known as the relative entropy of p with respect to q — note the reversal of emphasis).

For discrete $\displaystyle{ p }$ and $\displaystyle{ q }$ this means : $\displaystyle{ H(p, q) = -\sum_x p(x)\, \log q(x). \! }$ The situation for continuous distributions is analogous. We have to assume that $\displaystyle{ p }$ and $\displaystyle{ q }$ are absolutely continuous with respect to some reference measure $\displaystyle{ r }$ (usually $\displaystyle{ r }$ is a Lebesgue measure on a Borel σ-algebra). Let $\displaystyle{ P }$ and $\displaystyle{ Q }$ be probability density functions of $\displaystyle{ p }$ and $\displaystyle{ q }$ with respect to $\displaystyle{ r }$ . Then : $\displaystyle{ -\int_X P(x)\, \log Q(x)\, dr(x) = \operatorname{E}_p[-\log Q]. \! }$ NB: The notation $\displaystyle{ H(p,q) }$ is also used for a different concept, the joint entropy of $\displaystyle{ p }$ and $\displaystyle{ q }$