Frames, Surfaces, Reading

 

Proust and the Squid is a classic analysis of reading. Reading is a learned activity that involves nested multi-tasking between 

• Character recognition
• Word recognition
• Sentence parsing 
• Context detection
• Imagination - the oldest form of Virtual Reality
• Criticism 

Both Hawkins' Frame theory and Surfaces and Essences analogy-based theory is relevant. I lost the actual hard copy but finally remembered the title and bought the book for Kindle.

The Hawkins Frame - 1

In the long run, Hawkins abandons his initial description of "Frame" and (to my mind) wanders into simple Object Oriented Design and Mathematical constructs that lie behind Quantum Mechanics. I'm not sure gets this, but he starts with a frame-based description of consciousness, which is a fine example of putting this phenomenon on the test bench. Many philosophers claim this is not possible. Daniel Dennet is happy to allow what we say about our experience to be evidence of what that experience is. The problem with Dennet is that he was writing decades before Hawkins used the powerful frame language to speak of what it "feels like" to be in the world. Armed with Hawkins' way of speaking, it may be a good idea to loop back and read Dennet again.

With apologies to Hawkins, this is what I think a frame is:

• It is a frame of reference like Cartesian space
• It contains objects, each of which has a location in the space - for example, (x,y,z) coordinates.
• In object-oriented terms, coordinates are attributes of the object within the space. Other attributes include orientation (such as an (x,y,z) vector).
• Although it is not entirely clear, it seems that these objects also have a "velocity" vector - they may be moving through the reference frame.
• The observer occupies a position in the framework and has a complex set of objects that can be used to interrogate objects in the space. In a sense, the observer experiences this set of objects as "self". An example would be fingers exploring his beloved coffee cup. We will call these objects "sensors".
• A key element of Hawkins' account is the interaction between sensors and objects in the space to determine (or guess) the attributes of the objects. This interrogation results in an updated version of the frame itself - as modeled by the brain.
• All this is experienced dynamically - there is time in Hawkin's frame. When we speak of our experience, we call it "now", but "now" is always changing. This is the flow of time. We can be grateful for any model that makes an attempt to explain this most important aspect of the experience.
• Unlike an abstract mathematical "object", a frame is an experience. When we describe it as a mathematical object, we rely on analogy. This is the entire point of the Hawkins "frame". It is claimed to be the fundamental unit of experience, not the actual "God's Eye" view of the world, nor an abstract description of the world "as it is". It is a mental model, presumably implemented with neurons and synapses. As far as I know, this is the only attempt to formalize experience itself.

Some elements here remind me of Einstein's insight - particularly placing the observer in space and making observations relative to the frame of reference. In both cases, there is no "God's eye" view of the space, only the observations of the observer and the conclusions the observer reaches, such as the position and velocity of objects in the frame of reference. If you are familiar with Einstein's view of the world, Hawkins' "frames" are easy to picture.

Along the same line, Einstein was interested in the ability to interrogate objects to determine their properties. It turns out that it is not possible to learn certain pairs of properties exactly, say a and b. Precision in the measurement of a loses precision in b. Worst yet, counterintuitive, certain objects' properties don't exist until they are measured. But that is a different subject.

The point is that Einstein's mental picture of the Universe is a frame. A very powerful one. What it lacks is a sense of "now". Time is just another dimension, like the x,y,z of space.

Hawkins is shy about generalizing this model - sticking to our familiar 3-dimensional world. He could have escaped this a little by referring to the attributes of an object, which can be more or less unlimited. It would have been clearer (to me at least) if his model included the interrogation of the object, returning another frame since the object itself has all the attributes of a reference frame. Certain sensors are limited to certain queries - "touch" returns information that "look at" does not.

Anyone familiar with Object-Oriented design would wish to add (at least) object states which add some complexity to the result coming back from a query sensor. Other things like the concept of "interface" would be helpful to isolate the "experience" of the object from any secret and unobservable properties that may or may not produce the experience. The whole of OOD* seems applicable, and for good reason: OOD* has been developed precisely to describe (at least) the real world.

That said, there are aspects of the world where OOD doesn't seem very helpful, such as chemistry, cosmology, logic, mathematics, and Quantum Mechanics. Hawkins runs into these limitations since he claims his theory covers all we can know. His theory is only a re-invention or application of OOD - a special case. OOD covers a lot more of what we can know. It underlies the design of virtually all modern computer systems, including the Internet. But we must turn to other ways of "knowing" when leaving the familiar 3-dimensional world described in Hawkins' examples.

To be fair, the areas of knowledge where the Hawkins frames start to break down are precisely those that many find difficult to understand. We are also left with the biggest and most urgent things we need to understand: human-to-human interactions.

--------

* OOD = Object-Oriented Design

Frames All The Way Down

I have been rather dismissive of Hawkins' "Thousand Brains" idea but I can't help thinking he's on to something, specifically:

• Once you broaden your understanding of what a "frame*" is, it is quite plausible that people think in frames. The most powerful computer system we have built so far, the Internet, runs on a relatively simple "frame language", understood by over a billion "hosts" including the one you are reading this on.
• Hawkins provides a plausible description of human experience in terms of "frames" somehow being the unit of experience.
• It is plausible that the neocortex has some kind of structure that implements frames, including the ability to create frames "on the fly" and use them as a unit of memory and reasoning.
• Thousand Brain is not too dissimilar from the theory presented in Surfaces and Essences, which presents analogy as the "fuel of thought". "Surfaces" is really talking about language, which is a brain-to-brain communication device.  Theories of language, such as Chomsky's use frames to develop a theory of how language works.

Hawkins gets tripped up trying to describe complex frames, especially the ones that don't admit to a spatial interpretation. Object-Oriented Design has a rich language that applies directly. Starting at the simplest example, Hawkin's coffee cup, we can speak of objects as "frames". Object-Oriented concepts are extremely powerful in representing the real world - one reason these concepts underlie the Internet and some of its fundamental tools, such as Java.

Hawkins gets into trouble when he starts to speak as a neuroscientist, which he is not. He doesn't get published in Scientific Journals. In spite of his extensive knowledge of the subject, his ideas about how, exactly, columns in the neocortex create frames are sketchy, to say the least. But even here, he is pointing to a structure that is replicated all over the neocortex (the famous thousand brains). If we assume that we think in frames we may be forgiven to think that some structure in the brain is involved. Due to the enormous parallelism and recursiveness in experience, we would expect a large number of similar structures, not some kind of "organ".

Hawkins is particularly interested in the predictive nature of consciousness. I think he could be challenged on this. The brain is obviously filling in what it "expects" to be there but I am not quite sure we need the "priming" idea. It seems the brain "sees" what it expects to see in a "frame". What needs to be explained is how effortlessly it does this and how we are alerted to aspects of the real frame that are out of place. We know that this mechanism often fails, as in the phenomenon of "Change Blindness". In fact, it seems that change blindness may be the Achilles Heel for this aspect of Hawkins' theory. On the other hand, answering this question could turn out to provide strong extra evidence for the theory. 

Many applications in the Internet are at least attempting to be "predictive" - filling in details before the query is completed. "Type Ahead" is something we all experience. Hawkins seems to think that this is something "AI" can't do.

My thinking on this subject rides off in several directions.

• How it is that certain "virtual" environments, such as Second Life, feel so "real". Hawkins' ideas about felt reality are applicable.
• How do frames work in non-spacial domains, such as language, physics, and mathematics?
• Is the Universe actually built of frames or does it just look that way?
• Hawkins' book is very light on visual examples that would show the nested and recursive nature of the frames he is talking about. I'd like to provide some examples and perhaps shed some doubt on the idea that a "frame" is manifested in a column of the neocortex. 
• I'd like to present an alternative interpretation of the neurological "facts" in Hawkin's book. Specifically, I think a "frame" corresponds to a network of active neurons in the brain. There is an interesting visual example in Second Life, where a "frame" (a specific location in the world) "rezzes" a bit at a time, providing a way to think of how a frame could resolve itself almost instantly even though it is full of nested frames.

=================

* I use "frame" to refer to Hawkins' "reference frames"

See also

Object Oriented Design

The Languate of Information Science

Your Brain is a Virtual Reality Machine

 

Our ability to experience "virtual" worlds is not something new. Evolution has provided it as basic equipment along with our opposing thumbs. It's not a bug. It's a feature.

On the most fundamental level, we are constantly experiencing the future and the past: the flow of time. This ability is essential to what we call purposeful behavior. We love to debate whether or not other animals have this ability and to what degree, but it's clear that we have it in spades. It is essential to who we are as humans.

I am fascinated by how the brain produces the virtual world that we call "consciousness". It turns out to be quite a trick.

• Sense information arrives at different times. For example, sound and smell arrive with very little processing, but virtual information goes through massive complex analysis. Yet we experience it all at the "same time".
• We experience the decision to move after the brain has sent signals to move.
• Our experience slightly precedes what is going on in the real world - it is a prediction of what is happening. In the words of the Great One: I skate not to where the puck is but where the puck will be.

Careful observation of the phenomenon we call "consciousness" provides endless examples of how we are constantly bombarded with stunningly immersive "virtual" experiences. The smell of apple pie. The voice of a loved one on the phone. A spider crawling on our leg. Our brains instantly and effortlessly supply "virtual" senses to "enhance" the smallest trickle of "real" data.

On top of this, we structure reality into stories and "explanations". We are the story-telling ape. The good stories are embellished and handed on over generations, eventually knitting themselves into systems called "mythology" and "religion". We are quite willing to believe that we live on after death or that there is an old man in the sky behind everything that happens. We love our stories so much that we forget they are "just" stories.

Our brains are made to be fooled. We don't need computers to be plunged into an alternate reality. We can open a book and read: "Call me Ishmael" and suddenly we are whalers. We read "In the beginning" and we walk in the garden with Adam and Eve. If I may be allowed to speculate a bit, it seems that hearing these words rather than reading them creates a more powerful image. In the words of Bishop Pike: "What is too silly to be said can be sung". This points out one essential feature of how we experience virtual worlds: sometimes simplicity is an asset. And worlds that provide auditory input (especially speech) are readily accepted as real. We experience merely visual "worlds" (like a photograph) as "representation" but we are not "immersed".

Finally, in passing, I should note that virtual worlds created by written text must depend on the skill of the reader. Evolution does not provide this skill. It is learned. People vary in their ability to acquire this skill. Similarly, people vary in their ability to be "immersed" in virtual worlds. To some, the computer-generated worlds of VR remain cartoons. To others, they are instantly "real".

Modern "Virtual Worlds", if they are interesting at all, involves a learning curve that can be almost as daunting as learning to read. This aspect of things is often overlooked by prophets of the "metaverse". People are too lazy, preoccupied, or busy to learn a VR application that is more complicated than, say, Zoom. Those who developed VR "superpowers" will always be a tiny minority.

----------------

P.S. "A Thousand Brains" provides a deep and powerful framework for understanding how the brain creates the rich experience we call consciousness. In particular, it doesn't make a lot of difference where the sensations are coming from. Whatever we know about the real world (in a deeply recursive way) is applicable to another "world" where the sensations are coming from, say, a computer screen.