4.1.1 Information Theory And Shannon’s Information Capacity Theorem

The foundation for channel coding lies in information theory, which quantifies information and establishes limits on reliable communication through noisy channels. Information theory was developed by Claude Shannon (1916–2001) in the 1940s and 1950s.

As we saw in Section 3.3, the information content of a message is defined as:

I = \log_{2} \frac{1}{p} (bits)

(4.1)

where p is the probability of occurrence of the message. A less probable message conveys more information than one that occurs frequently.

The total information, I_t, contained in a set of M messages is the sum of the information in the individual messages. That is:

I_{t} = \sum_{i = 1}^{M} p_{i} M \log_{2} \frac{1}{p_{i}} (bits)

(4.2)

The average information content per message is the entropy, E, which is defined as:

E = \frac{I_{t}}{M} = \sum_{i = 1}^{M} p_{i} \log_{2} \frac{1}{p_{i}} (bits per message)

(4.3)

If r messages are transmitted per second, the average information rate, R, is therefore:

R = r E (bps)

(4.4)

The maximum entropy of an M-message set is log₂M, which occurs when the transmitted messages are equally likely to occur.

Shannon’s information-capacity theorem (also known as the Shannon–Hartley law) establishes the maximum achievable data rate, H, of a band-limited channel impaired by additive white Gaussian noise (AWGN):

H = B \log_{2} (1 + \frac{C}{N}) (bps)

(4.5)

where B is the channel bandwidth in hertz, and C/N is the carrier-to-noise ratio at the input of the receiver (C is the average transmitted (or received power) in watts; and N is the average noise power in watts).

Re-writing in terms of the noise power spectral density (N = N₀B):

H = B \log_{2} (1 + \frac{C}{N_{0} B}) (bps)

(4.6)

where N₀ is the single-sided noise power spectral density in watts/hertz.

For an ideal digital link transmitting at the channel capacity, H=1/T_b=R_b, where T_b is the bit duration, and R_b is the bit rate in bits per second. The energy per bit, E_b, is:

E_{b} = C T_{b} = \frac{C}{H}

(4.7)

Dividing both sides by N₀:

\frac{E_{b}}{N_{0}} = \frac{C}{H N_{0}}

(4.8)

And substituting into Equation (4.6) gives:

\frac{H}{B} = \log_{2} (1 + \frac{E_{b}}{N_{0}} \frac{H}{B})

(4.9)

Re-arranging Equation (4.9) in terms of the E_b/N₀ratio:

\frac{E_{b}}{N_{0}} = \frac{2^{\frac{H}{B}} - 1}{\frac{H}{B}}

(4.10)

For a system that is not ideal, H ≠ R_b, and H is replaced with R_b in Equation (4.9).

Figure 4.1 shows the relationship of Equations (7.9) and (7.10) in terms of the bit rate, R_b—that is, the figure plots R_b/B versus E_b/N₀. The ratio of the bit rate to the bandwidth (R_b/B) is called the bandwidth efficiency or the spectral efficiency, which warrants a little further investigation.

Figure 4.1. Relationship between R_b/B and E_b/N₀.

As bandwidth approaches infinity, E_b/N₀ tends toward ln(2)=0.693, or –1.6 dB. This is the Shannon limit, which sets the lower bound on the E_b/N₀ required for error-free communication on a single channel, regardless of modulation or coding.

Note also that the curve in Figure 4.1 defines the points where R_b=H. Operations in the region where R_b<H have the potential to support error-free transmission. Transmission in the region R_b>H is not possible without error.

Two broad operating regimes are commonly distinguished:

Power-limited systems (where R_b<B), in which performance is constrained primarily by available signal power. In satellite communications, for example, the long transmission distance, atmospheric effects, rain attenuation, and limited transmitter power combine to produce a strongly power-limited link. Channel coding is therefore essential to achieve reliable communication at acceptable power levels.
Bandwidth-limited systems (where R_b>B), in which performance is constrained primarily by available bandwidth. Examples include optical fiber systems, cable modems, and high-capacity microwave links. In fiber-optic systems the channel is typically bandwidth-limited rather than power-limited, yet coding remains essential to support extremely high spectral efficiencies.

Thus, channel coding provides a systematic method for reducing the probability of error by anticipating the most probable error patterns and introducing structured redundancy to correct them. It does not eliminate errors entirely, but it can reduce their probability to arbitrarily low levels within the constraints imposed by bandwidth, power, latency, and implementation complexity.

Modern coding techniques—such as convolutional codes (Section 4.6), turbo codes (Section 4.10.1), low-density parity-check (LDPC) codes (Section 4.10.2), and polar codes (Section 4.10.3—enable performance within approximately 1 dB (and in some cases less) of the theoretical bound.

While the capacity formula defines an absolute upper bound on achievable data rate for a given bandwidth and signal-to-noise ratio (SNR), it does not by itself describe how such performance may be approached in practice. The question of whether suitable coding schemes exist—and under what conditions reliable communication is possible—is addressed by Shannon’s second major result, the channel-coding theorem, discussed next.