19 Classical and quantum entropy
19.1 Classical entropy
In this chapter we will generalize classical and quantum concepts of entropy and take a deeper look at their relation to information. We begin with the classical entropy H. This was introduced in Chapter 1 as a practical measure of the amount of information in a message. If we consider a source that generates M equally likely messages, we define the entropy H to be
This definition has a number of intuitively appealing properties. For example, if two messages are independent, ...