Question from Hartely's "Transmission of Information".

First a quick summary of the beginning of the article:

Let [; s ;] be a number of symbols from which we can make a selection and let [; n ;] be the number of selections from these symbols. Then the number of possible sequences involving [; n ;] selections is [; s^{n} ;].

But we want to define the amount of information in a given symbol sequence as proportional to the number of selections made as opposed to the number possible sequences. So let [; H ;] be the amount of information in a given sequence, and let [; K ;] be a constant of proportionality so that the information in a given sequence is given by

[; H = Kn ;]

where [; K ;] is dependent upon the number of possible symbols [; s ;].

We want the amount of information in two different symbol systems to be the same when the number of possible sequences is the same. That is, when

[; s_{1}^{n_{1}} = s_{2}^{n_{2}} ;]

we want

[; H = K_{1}n_{1} = K_{2}n_{2} ;]

where [; K_{1} ;] is dependent on [; s_{1} ;] and [; K_{2} ;] on [; s_{2} ;]. From this it follows that

[; \frac{K_{1}}{\log s_{1}} = \frac{K_{2}}{\log s_{2}} ;].

It's at this point I have a question, which I figure is pretty basic. The article states that this relation holds for all [; s ;] only if [; K = K_{0}\log s ;] where [; K_{0} ;] is the same for all systems. Could someone elaborate on this?

From this point we can get to the intuitive idea that

[; H = \log s^{n} ;]

and I don't have trouble with that step. It's just the basic math in between I'm not seeing.

Posted Apr 23, 2015, 6:25 PM UTC

12
over_the_lazy_dog•

Let [; c ;] be some constant. Suppose it's the constant for some arbitrary system of your choosing (defined once for all time, with [; s_0 ;]).

Now for any other system, you have: [; \frac{K_1}{\log{s_1}}=\frac{c}{\log{s_0}} ;].

Now just multiply both sides by [; \log{s_1} ;] and define [; K_0 ;] to be [; \frac{c}{\log{s_0}} ;].

2
EntropyAnimalsOP•

Thank you much, this clears it up. I could see the logic of the algebra, but I was stuck on the justification. Just a math maturity thing.

1