3.3.1 Information And Entropy

To quantify information content, consider how unexpected a message is. The information I conveyed by a message is given by:

I = \log_{2} (\frac{1}{P}) bits

(3.6)

where P is the probability that the message will be transmitted. Messages that occur more frequently carry less information.

Suppose a source transmits one of four weather conditions—Fine, Cloudy, Rain and Snow—which are not equally probable. Figure 3.10 lists their probabilities, corresponding information content, and example binary representations.

Figure 3.10. Example messages, probabilities, and PCM codewords.

This average information of the source is known as its entropy is:

H = \sum_{i = 1}^{M} P_{i} \log_{2} (\frac{1}{P_{i}}) bits/message

(3.7)

where M is the number of different messages produced by the source and P_i is the probability of transmitting the i^th message. For the example in Figure 3.10, the entropy of the source is:

H = 1.15 \times 0.45 + 2 \times 0.25 + 2.74 \times 0.15 + 2.74 \times 0.15 = 1.84 b i t s / m e s s a g e

(3.8)