Question from Hartely's "Transmission of Information".
First a quick summary of the beginning of the article:
Let [; s ;] be a number of symbols from which we can make a selection and let [; n ;] be the number of selections from these symbols. Then the number of possible sequences involving [; n ;] selections is [; s^{n} ;].
But we want to define the amount of information in a given symbol sequence as proportional to the number of selections made as opposed to the number possible sequences. So let [; H ;] be the amount of information in a given sequence, and let [; K ;] be a constant of proportionality so that the information in a given sequence is given by
[; H = Kn ;]
where [; K ;] is dependent upon the number of possible symbols [; s ;].
We want the amount of information in two different symbol systems to be the same when the number of possible sequences is the same. That is, when
[; s_{1}^{n_{1}} = s_{2}^{n_{2}} ;]
we want
[; H = K_{1}n_{1} = K_{2}n_{2} ;]
where [; K_{1} ;] is dependent on [; s_{1} ;] and [; K_{2} ;] on [; s_{2} ;]. From this it follows that
[; \frac{K_{1}}{\log s_{1}} = \frac{K_{2}}{\log s_{2}} ;].
It's at this point I have a question, which I figure is pretty basic. The article states that this relation holds for all [; s ;] only if [; K = K_{0}\log s ;] where [; K_{0} ;] is the same for all systems. Could someone elaborate on this?
From this point we can get to the intuitive idea that
[; H = \log s^{n} ;]
and I don't have trouble with that step. It's just the basic math in between I'm not seeing.
Posted Apr 23, 2015, 6:25 PM UTC
Let [; c ;] be some constant. Suppose it's the constant for some arbitrary system of your choosing (defined once for all time, with [; s_0 ;]).
Now for any other system, you have: [; \frac{K_1}{\log{s_1}}=\frac{c}{\log{s_0}} ;].
Now just multiply both sides by [; \log{s_1} ;] and define [; K_0 ;] to be [; \frac{c}{\log{s_0}} ;].
Thank you much, this clears it up. I could see the logic of the algebra, but I was stuck on the justification. Just a math maturity thing.