Seite - 26 - in Applied Interdisciplinary Theory in Health Informatics - Knowledge Base for Practitioners

Performing such a minimisation, of course, presupposes that we do have available some measure of information content. This is the topic of the next section. 2. Information content and entropy Before introducing a measure of information content, let us first explore a simple example to motivate the precise choice of measure. This is a necessarily brief introduction. A next step for the interested reader may be to read a more extended tutorial such as that provided in [11]. Consider an array of N binary switches, where N could be any integer greater than zero. For each switch, we have two possible states. One could think of these as “on” and “off”. Correspondingly, we have two possible messages: one indicating the switch is in state “on” and one indicating the switch is in state “off”. With just one switch, we can store 1 “bit” of information: we just need to receive one message in order to determine the state of that switch. We can store 2 bits of information with an array of two switches, and we will require 2 messages each of 1 bit or one message of 2 bits to determine the state of that array. Note that our measure of information content is additive; that is, the information content of a single message from an array of 2 binary switches is simply the sum of the information content of single messages sent from each of those switches separately. In general, with an array of N switches, we will need a message that is N bits long in order to determine the internal state of that array of switches. Now, let us also look at the total number of states of an array of binary switches. For two switches, we have four possible states: {on, on}; {on, off}; {off, on}; {off, off}. Correspondingly, we will have four possible messages that will tell us the state of the array in a single message. In the general case of N switches in an array, we have 2N possible states and 2N possible messages. Now, if we assume that each switch acts independently, and that each of these possible messages is equally likely, then any one message has a probability p = 1/(2N) of occurrence. Consider an outcome in which a message m of length N is received. The above discussion motivates a requirement for a measure of information content that is additive. In addition, we have seen that the number of states in a system tends to increase exponentially. This suggests the use of a logarithmic function such as that of equation 1: Eq 1. ݄ሺݔሻൌ െ݈݋ ݃ଶ݌ ሺݔሻ Substituting our message m with probability of occurrence p(m) = 1/(2N) into Equation 1, we get: ݄ሺ݉ሻൌ െ݈݋ ݃ଶቀͳ ʹேൗ ቁൌ ݈݋ ݃ଶሺʹ ேሻൌܰ Equation 1 is thus returning us our informally proposed measure of information content; it is in fact the definition of the Shannon information content of an outcome. We now need to generalise this. When a patient presents, that patient’s state is not known with certainty. Thus, the possible messages that may be received form an ensemble M, with each message ݉אܯ having a probability of occurrence, p(m). We use the word ensemble here in a statistical sense. Writing this out more formally, M is a P.Krause / InformationTheoryandMedicalDecisionMaking26