Home-> Concepts->Entropy and Information

October 31, 1999
Updated September 3, 2001
Updated October 27, 2011

Entropy and Information

The unit of information used in neural systems should explicitly be a function of time like the unit of hertz. For example, the information in the time-independent sequence of letters S = {a, b, c} is unique for the object it represents for the entire duration that it exists. However, in a dynamical systems, the potential information in a sequence can be smaller or larger depending on the forces acting on the system. The unfolding of a sequence depends on the environment or system that the sequence represents.

Temporal Sequences and the Binary Interval

In the real physical world, as compared against a mathematical description of it, information is essentially contained in the states of matter or energy. Entropy is a descriptive attribute of the states of energy (or matter). In this very general context, information exists or emerges from within the states of energy.

Information can also be described mathematically. Information as an abstraction can be spoken and written of mathematically using a function describing the unfolding of a binary sequence which is the logarithm of base 2.

is the number of possible events or outcomes of the system. Entropy described mathematically is confined within the general equation above.

Mathematically speaking, the unit of information is the bit: either 0 or 1. The bit represents a time-independent unit of information. But information can be defined as an elemental binary unit which changes as a function of time: f({0,1}; t).

We could also measure information change more naturally as the rate-of-change of the information or bytes in a system. The frequency of a system, measured in hertz, is the rate-of-change of the oscillations in a system. So a system vibrating at 1 hertz means a bit is changing from 0 to 1 and back again in 1 second. Visualize the movement of 1 cycle of a sine wave. Now a sequence of notes on a sheet of music represent objects whose units are in hertz. But observe that notes on a music sheet represents a frequency that is constant in time. The representation of notes is in a subclass of the type of dynamic sequence described above.

Thoughts Are Continuous

Our thinking processes are the result of real physical or biological events occurring as a function of time. Our thinking processes exists as a real dynamical energy system. It is not a mathematical abstraction which is not real in the sense of having to exist in the physical world. We can think about mathematical processes, but these abstraction themselves do not have a real physical existence. This is in contrast to our thoughts which are created because of the real energetic neural synapses in our brains.

Our thoughts are created by real physical processes which are subject to the constraints of the Heisenberg uncertainty principle which limits the creation of our thoughts as a function of time. However, what we think is unlimited because what we think is not physically real.

Temporal sequences are intrinsically multi-dimensional in nature [1]. Information in the mind is created dynamically as a temporal sequence. The thoughts in our mind seem fleeting because they are generated or created everytime our neural waves move across parts of the brain. The action of thinking means that our brain creates a temporal sequence of energy patterns that only lasts for a very short duration. In order to hold a thought our brain needs to continually recreated these sequences of energy patterns or synapses.

In trying to create a model for the software tools I had try to try to rid myself of the old way of thinking about information. When the mind generates information, it only lasts in the order of a fraction of a second. Then it needs to be recreated. I'm going on faith from past physiological studies that the physical structures which produces these energy patterns are the coherent, synapsing neurons. In the software model, I've try to separate the local micro-ensemble effect of neurons from those global clustering effects in synchronous neural circuits, and then try to put the pieces back together.


[1] Dennis Gabor asked [2], "... what it is that prevents any instrument from analysing the information area with an accuracy of less than a half unit. The ultimate reason for this is evident. We have made of a function of one varialbe -- time or frequency -- a function of two variables -- time and frequency." He said that is the mathematical identity which is at the root of the fundamental principle of communication. He said, "We see that the r.m.s. duration of a signal, and its r.m.s. frequency-width define a minimum area in the information diagram."

Further along, in section 4 of Dennis Gabor's Theory of Communication, he says, "Moreover, it suggests that it might be possible to give a more concrete interpretation to the information diagram by dividing it up into "cells" of size one half, and associating each cell with an "elementary signal" which transmitted exactly one datum of information."

[2] Theory of Communication, Dennis Gabor, 1946,
The Journal of the Institution Of Electrical Engineers, 93(3):429-457.

next:   Waves