Entropy of a probability distribution
The entropy of a probability distribution is a number that describes the degree of uncertainty or disorder the distribution represents.
Assume we have a set of two mutually exclusive propositions (or equivalently, a random experiment with two possible outcomes). Assume all two possibilities are equally likely.
Then our advance uncertainty about the eventual outcome is rather small - we know in advance it will be one of exactly two known alternatives.
Assume now we have a set of a million alternatives - all of them equally likely - rather than two.
It seems clear that our uncertainty now about the eventual outcome will be much bigger.
- Given a discrete probability distribution function f, the entropy H of the distribution (measured in bits) is given by
- Given a continuous probability distribution function f, the entropy H of the distribution (again measure in bits) is given by
Note that some authors prefer to use other units than bit to measure entropy, the formulas are then slightly different. Also, the symbol S is sometimes used, rather than H.
- Entropy in thermodynamics and information theory
- Discrete probability distribution
- Continuous probability distribution
- Probability theory