Post by speakertoanimals on Jan 28, 2011 13:08:21 GMT 1
I could give you YET AGAIN the quote from the Shannon paper, that says probabilities have to be computed from the ensemble of all possible messages, NOT just for the particular message you are going to send.
You are still misunderstanding the basics of information theory.
Now as regards your point about CHANGE. Suppose we tried to use that strategy to transmit the result of N coin tosses from a fair coin.
SO, we would first have to say what the first toss was H or T. Except let's suppose we don't actually care, and we will be treating HTT to be the same as THH -- it's only 1 bit overall anyway.
So, change or no change? This is still a BINARY state that needs to be transmitted. 0 or 1 again, except that 0 now means change (why not? I can encode it however I like!), and 1 means no change.
Except for a coin, the state of change (next toss is different from the previous one) is AS LIKELY (over the ensemble of all possible coin toss sequences) as the state of no change.
Hence we find, yet again, that the information content in a string of N coin tosses is given by 1 bit for initial H or T, then N-1 bits for change/no change. We hence make no gain whatsoever compared to just sending H or T at each toss.
The ONLY case where we can make a gain is where we have correlation, and where a long sequence of no change is more probable than we might expect given a totally random process. This is the case in telephone communication, which is probably why you are under the mistaken impression that it is the ONLY case.
ANd yet again, I refer you back to the quote from SHannon -- perhaps you would care to explain WHY your computation of the information content of a stirng of zeros was correct given that quote?
Except you can't! you're just plain wrong, don't have the intelligence or the balls to see why you are wrong, and admit you were wrong and learn something.
I pity your poor blody students, they aren't getting taught information theory properly.
The point about a string of zeros containing as much information as any other random string of 0s and 1's is an important one, because it is WHY some people have triede to go beyind Shannons definition of information content, pointing out precisely that a repeated string we intuitively feel contains less information. I quote:
homepages.cwi.nl/~paulv/papers/info.pdf
Page 11, end of section 2.1
As regards the point about the probabilioty from the source (as opposed to what you did which was the probabilioty FROM the MESSAGE, the same paper states (Page 2, first paragraph):
And the same paper then quotes SHannon again:
Which is what I said before, probability from source, NOT from particular message, according to Shannon.
QED and all that, and you can kiss my shiny metal arse..................
You are still misunderstanding the basics of information theory.
Now as regards your point about CHANGE. Suppose we tried to use that strategy to transmit the result of N coin tosses from a fair coin.
SO, we would first have to say what the first toss was H or T. Except let's suppose we don't actually care, and we will be treating HTT to be the same as THH -- it's only 1 bit overall anyway.
So, change or no change? This is still a BINARY state that needs to be transmitted. 0 or 1 again, except that 0 now means change (why not? I can encode it however I like!), and 1 means no change.
Except for a coin, the state of change (next toss is different from the previous one) is AS LIKELY (over the ensemble of all possible coin toss sequences) as the state of no change.
Hence we find, yet again, that the information content in a string of N coin tosses is given by 1 bit for initial H or T, then N-1 bits for change/no change. We hence make no gain whatsoever compared to just sending H or T at each toss.
The ONLY case where we can make a gain is where we have correlation, and where a long sequence of no change is more probable than we might expect given a totally random process. This is the case in telephone communication, which is probably why you are under the mistaken impression that it is the ONLY case.
ANd yet again, I refer you back to the quote from SHannon -- perhaps you would care to explain WHY your computation of the information content of a stirng of zeros was correct given that quote?
Except you can't! you're just plain wrong, don't have the intelligence or the balls to see why you are wrong, and admit you were wrong and learn something.
I pity your poor blody students, they aren't getting taught information theory properly.
The point about a string of zeros containing as much information as any other random string of 0s and 1's is an important one, because it is WHY some people have triede to go beyind Shannons definition of information content, pointing out precisely that a repeated string we intuitively feel contains less information. I quote:
Thus, we have discovered an interesting phenomenon: the description of some strings can be compressed
considerably, provided they exhibit enough regularity. However, if regularity is lacking, it becomes more
cumbersome to express large numbers. For instance, it seems easier to compress the number “one billion,”
than the number “one billion seven hundred thirty-five million two hundred sixty-eight thousand and three
hundred ninety-four,” even though they are of the same order of magnitude.
We are interested in a measure of information that, unlike Shannon’s, does not rely on (often untenable)
probabilistic assumptions, and that takes into account the phenomenon that ‘regular’ strings are compress-
ible. Thus, we aim for a measure of information content of an individual finite object, and in the information
conveyed about an individual finite object by another individual finite object. Here, we want the information
content of an object x to be an attribute of x alone, and not to depend on, for instance, the means chosen
to describe this information content. Surprisingly, this turns out to be possible, at least to a large extent.
The resulting theory of information is based on Kolmogorov complexity
considerably, provided they exhibit enough regularity. However, if regularity is lacking, it becomes more
cumbersome to express large numbers. For instance, it seems easier to compress the number “one billion,”
than the number “one billion seven hundred thirty-five million two hundred sixty-eight thousand and three
hundred ninety-four,” even though they are of the same order of magnitude.
We are interested in a measure of information that, unlike Shannon’s, does not rely on (often untenable)
probabilistic assumptions, and that takes into account the phenomenon that ‘regular’ strings are compress-
ible. Thus, we aim for a measure of information content of an individual finite object, and in the information
conveyed about an individual finite object by another individual finite object. Here, we want the information
content of an object x to be an attribute of x alone, and not to depend on, for instance, the means chosen
to describe this information content. Surprisingly, this turns out to be possible, at least to a large extent.
The resulting theory of information is based on Kolmogorov complexity
homepages.cwi.nl/~paulv/papers/info.pdf
Page 11, end of section 2.1
As regards the point about the probabilioty from the source (as opposed to what you did which was the probabilioty FROM the MESSAGE, the same paper states (Page 2, first paragraph):
In the Shannon approach, however,
the method of encoding objects is based on the presupposition that the objects to be encoded are outcomes
of a known random source—it is only the characteristics of that random source that determine the encoding,
not the characteristics of the objects that are its outcomes.
the method of encoding objects is based on the presupposition that the objects to be encoded are outcomes
of a known random source—it is only the characteristics of that random source that determine the encoding,
not the characteristics of the objects that are its outcomes.
And the same paper then quotes SHannon again:
Frequently the messages have meaning; that
is they refer to or are correlated according to some system with certain physical or conceptual
entities. These semantic aspects of communication are irrelevant to the engineering problem.
The significant aspect is that the actual message is one selected from a set of possible messages.
The system must be designed to operate for each possible selection, not just the one which will
actually be chosen since this is unknown at the time of design
is they refer to or are correlated according to some system with certain physical or conceptual
entities. These semantic aspects of communication are irrelevant to the engineering problem.
The significant aspect is that the actual message is one selected from a set of possible messages.
The system must be designed to operate for each possible selection, not just the one which will
actually be chosen since this is unknown at the time of design
Which is what I said before, probability from source, NOT from particular message, according to Shannon.
QED and all that, and you can kiss my shiny metal arse..................