Citizendium - a community developing a quality comprehensive compendium of knowledge, online and free. Click here to join and contribute—free |

CZ thanks our previous donors. Donate here. Treasurer's Financial Report -- Thanks to our content contributors. -- |

# Entropy of a probability distribution

### From Citizendium, the Citizens' Compendium

The **entropy** of a probability distribution is a number that describes the degree of uncertainty or disorder the distribution represents.

## Contents |

## Examples

Assume we have a set of two mutually exclusive propositions (or equivalently, a random experiment with two possible outcomes). Assume all two possibilities are equally likely.

Then our advance uncertainty about the eventual outcome is rather small - we know in advance it will be one of exactly two known alternatives.

Assume now we have a set of a million alternatives - all of them equally likely - rather than two.

It seems clear that our uncertainty now about the eventual outcome will be much bigger.

## Formal definitions

- Given a discrete probability distribution function f, the entropy H of the distribution (measured in bits) is given by
- Given a continuous probability distribution function f, the entropy H of the distribution (again measure in bits) is given by

Note that some authors prefer to use other units than bit to measure entropy, the formulas are then slightly different. Also, the symbol S is sometimes used, rather than H.

## See also

- Entropy in thermodynamics and information theory
- Discrete probability distribution
- Continuous probability distribution
- Probability
- Probability theory