>

From Uncertainty to Information

Now that we have defined uncertainty and given it a unit of measure, we shall put information in terms of uncertainty. Let us examine our boxes with the blue ball again, but for simplicity, let's say there are just eight boxes. At the start, before the ball's location is known, there is an uncertainty of 3 bits ( $\log_2 8 = 3$) [equations (2) and (13)]. After the ball has been located, there is no uncertainty as to where the ball is. After all, you are staring right at it. Since your uncertainty has gone from 3 bits to 0 bits, you have GAINED 3 bits of information.

Now suppose we play the guessing game, but that I refuse to tell you the answer to the last yes-no question you ask. Before playing the game your uncertainty is 3 bits, but after the game is over, you still don't know which box the blue ball is in. Because you are still uncertain by 1 bit, you learned from me only 3 - 1 = 2 bits. In other words, the uncertainty (H) before the input is received minus the uncertainty after it is received is the information you gained (R):

 

H_before - H_after = R . (15)

  

Figure 2: Many sequence logos printed on top of each other.

There were 12 lambda cI and cro binding sites, 8 lambda O binding sites, 58 CRP binding sites, 34 ArgR binding sites, 38 LexA binding sites, 8 TrpR binding sites and 12 AraC binding sites. See [15].

  

Figure 3: Sequence logo of words in the English language.

Words from the Unix dictionary, /usr/dict/words, were aligned by their first letter. Vowels are darker and consonants are lighter. In color versions of this paper, vowels are red, consonants are green and the letter Y is in blue.

  

Figure 4: Sequence logo of words in this paper.

Words were aligned by their first letter. In color versions of this paper, vowels are red, consonants are green and the letter Y is in blue.