What Is Entropy?
What Is Information Entropy?
Preview: Learn more about entropy and how it measures the amount of information contained in a source.
Entropy is a measure of the average amount of information produced by an information source. Introduced by the American mathematician and engineer Claude Shannon in 1948, entropy quantifies the uncertainty associated with a message before it is received. It forms one of the cornerstones of information theory and establishes the theoretical limit on how efficiently information can be represented or compressed without losing any data.
The fundamental idea behind entropy is simple: unlikely events convey more information than common ones. For example, if a weather forecast predicts sunshine every day during summer, hearing that tomorrow will also be sunny provides very little new information. However, hearing that tomorrow will bring snow would be highly unexpected and therefore conveys much more information. Entropy measures this average level of surprise across all possible messages produced by a source.
If every symbol produced by a source is equally likely, the uncertainty is high and the entropy is correspondingly large. Conversely, if one symbol occurs much more frequently than all others, there is less uncertainty because the next symbol can often be predicted correctly. Such a source has lower entropy because less new information is produced on average.
Shannon showed that the entropy of a source depends only on the probabilities of its symbols. For a source producing symbols with probabilities p₁, p₂, …, pₙ, the entropy is given by
and is measured in bits per symbol. This value represents the minimum average number of bits required to represent each symbol without losing information.
Entropy has profound practical significance because it establishes the theoretical limit of lossless data compression. No lossless compression algorithm can, on average, represent information using fewer bits than the source entropy. Compression techniques such as Huffman coding and arithmetic coding seek to approach this limit by assigning shorter codewords to more probable symbols and longer codewords to less probable ones.
A useful analogy is packing luggage for a journey. If every suitcase always contains exactly the same items, very little additional description is required because the contents are predictable. If every suitcase contains a completely different collection of items, considerably more information is needed to describe them. Entropy measures this average unpredictability of the source.
It is important to distinguish information entropy from thermodynamic entropy. Although Shannon borrowed the term because of mathematical similarities, the two concepts describe different phenomena. Thermodynamic entropy measures disorder in physical systems, whereas information entropy measures uncertainty in information sources.
Today, entropy underpins virtually every area of digital communications, data compression, cryptography, machine learning, and information processing. It influences the design of communication systems, determines the theoretical limits of compression, and provides the mathematical foundation for modern information theory. In essence, entropy answers one of the most fundamental questions in communications engineering: how much information does a message actually contain?
Back to reading