What is Information? The Ins and the Outs [Part 2]

Shannon's Information Theory vs. Tononi's Integrated Information Theory

Midjourney

Welcome to Part 2 in our two-part series on: What is Information? The Ins and the Outs.

Last week, in Part 1, we explored information according to Shannon’s Information Theory — which is a theory of information that is heavily used in the sciences, engineering, and artificial intelligence. If you haven’t done so already, you may want to read Part 1 first before reading this article, especially if you are new to Information Theory.

What is Information? The Ins and the Outs [Part 1]

In this article, we will explore information according to Giulio Tononi’s Integrated Information Theory (IIT) — a popular yet controversial theory of consciousness. You will notice both Information Theory and IIT use similar words when describing what they mean by information, but you will very quickly notice their conceptions of information are radically different.

In this article, we’ll slowly build the definition of information according to IIT, piece by piece. To do this, we will review four key characteristics of information according to IIT. Trying to explain the entire theory in one article is an impossible task. The explanation of IIT I give here is a highly simplified version with necessary omissions. We will explore the theory in more detail, and with a more critical eye, in a future article.

Before we begin, there are two claims we should know about and one thing we should keep in mind.

The first claim IIT makes is that consciousness exists. Consciousness is real in the sense that it is an irreducible phenomenon that cannot be eliminated or explained away, as some eliminative materialism theories claim. Consciousness is where IIT begins; It begins with phenomenology.

The second claim is that all conscious experiences have five essential properties—or axioms. (In scientific theories, axioms are simply the basic assumptions on which a theory is built or founded, i.e. they cannot be inferred from other statements.) One of the axioms of IIT is information. The other four axioms are intrinsic existence, integration, exclusion and composition. So, according to the latest version of IIT (version 4.0):

every experience is for the experiencer (intrinsically), specific (information), unitary (integration), definite (exclusion), and structured (composition)

In this article, we will be introduced to the other four axioms, but only insofar as they help us understand information according to IIT.

And the one thing we should keep in mind is that IIT is a theory about consciousness. The core idea is that consciousness is integrated information. So, information is always mentioned in relation to conscious experience. It is an assumed essential building block of consciousness.

1. Specific

The first key characteristic is that IIT claims information is specific.

Let’s unpack that.

If you think about all the conscious experiences you have ever had — each experience is unique — it is specifically this experience, not that experience.

The exact conscious experience you are having right now, you have never experienced before. You have never experienced reading precisely this combination of words, with the precise configuration of items in your background, sounds in your ears, and smells in the air. When you combine your current perceptual experience with your current emotional experience and memories, it’s clear that this conscious experience is different from all other possible conscious experiences you have had or ever will have.

It is this differentiation that makes consciousness informative. It is precisely because each conscious experience is specifically this experience and not that experience that makes each conscious experience informative.

However, specificity and differentiation alone do not fully capture what IIT means by information. There are three more characteristics we need to unpack, so let's continue putting the pieces together.

2. Differences that make a difference

The second characteristic draws on Gregory Bateson's definition of information:

[T]he world of form and communication invokes no things, forces, or impacts but only differences and ideas. (A difference which makes a difference is an idea. It is a ‘bit,’ a unit of information.)

—Bateson, 1972, p. 276

What Bateson means here is that information happens when a system differs, and those differences make a difference. Notice how it is only differences that make a difference that count as information.

IIT is a physical theory of consciousness. For biological systems — like humans — the differences that make a difference happen in the physical brain.

Currently, at this precise moment, your brain is in a particular configuration and state. For simplicity, we can think of this as the neurons in the brain being wired up in a particular way, with some neurons firing while others are not (the brain is more complicated than this, but this notion will work here). Let’s call this your brain’s current physical state.

Because IIT is a physical theory of consciousness, your brain's current physical state must differ from all other possible physical states so your current conscious experience differs from all other possible conscious experiences.

And, importantly, for our understanding of information — it is only the difference in the brain’s physical states that makes a difference to itself that counts as information. It’s the differences that make a difference that are informative.

This means IIT’s definition of information is more closely aligned with the etymology— the original, historical meaning — of information than it is with Information Theory.

The word information comes from the Latin informare, which means to give form to or to shape. Historically, it referred to the idea of giving form to the mind or imparting knowledge. So, information in IIT is about the brain’s ability to transform or shape itself in a way that is meaningful to the brain.

---

Measuring information

To calculate information, IIT adapts concepts from Information Theory, such as entropy and the transmission of information, but applies these concepts in unique ways.

Intrinsic information is measured in units called intrinsic bits, or ibits, to distinguish them from the bits used in Information Theory. An ibit corresponds to a point-wise information value measured in bits but is weighted by a probability.

While I don’t want to spend too much time on the math in this article, those interested in mathy things might find the following helpful (skip the section between the wavey lines if mathy-things is not your thing):

Whereas Information Theory primarily focuses on quantifying how much information is communicated, IIT adapts these principles to understand the causal relationships within a system. Specifically, IIT calculates probabilities to determine how one state of a system can influence another. This involves examining both forward and backward probabilities. The theory can predict how much a specific state leads to or results from another state; that is, it can measure its causal impact. Information is quantified using the following formulas (which use the logarithm of probability ratios, similar to Kullback-Leibler Divergence):

For effect information, forward probabilities calculate the increase in the likelihood of a future state 𝑠‾ given the current state 𝑠.

The formula for effect information:

\(\text{ii}_e(s,\overline{s}) = P_e(\overline{s} | s) \log_2 \left( \frac{P_e(\overline{s} | s)}{P_e(\overline{s})} \right)\)

Conversely, for cause information, the formula incorporates both backward and forward probabilities. The backward probability 𝑃𝑐←(𝑠‾∣𝑠) is calculated using Bayes' rule and reflects how past states influence our understanding of the current state. The forward probability in the logarithmic term calculates the increase in the probability of the current state given a past state.

The formula for cause information:

\(\text{ii}_c(s, \overline{s}) = P_c^\leftarrow (\overline{s} | s) \log_2 \left( \frac{P_c(s | \overline{s})}{P_c(s)} \right) \)

A more detailed explanation of these formulas is provided on pages 13 and 14 in the latest version of the IIT.

Okay, what does that all mean?

In short, according to IIT, information has what is called cause-effect power. This means information can make things happen and change things in the physical world.

There is a principle in physics called the causal closure of the physical, which asserts that a change in the physical world must have a physical cause. So, if information can make things happen and change things in the world, it must, by definition, be physical.

So, our working definition of information according to IIT is as follows:

a physical difference in a system (like a brain) that makes a meaningful physical difference.

We’re getting closer to what IIT means by information, but we have two more pieces to add.

3. The intrinsic perspective

The third characteristic of information, according to IIT, is that it is not only specific and physical; it is — intrinsic. It’s the difference that makes a difference from the perspective of the system itself.

This is probably the biggest dissimilarity between information according to IIT and information according to Information Theory.

Shannon Information views information from an extrinsic perspective; information is always assessed from the outside observer's perspective. It measures the statistical relationship between inputs and outputs (or unknown to known) rather than any meaning or significance an event might have. The sender and receiver of the inputs and outputs are assumed to understand the meaning, but Information Theory is only concerned with how accurately and efficiently a message is sent and received. It is not surprising, then, that Tononi calls Shannon information extrinsic information.

In last week’s article, we noted two things about information, according to Information Theory, that are important to highlight again here.

1. Observer-independent vs. observer-independent

Information, according to Information Theory, is observer-dependent. When we calculate the bits of information — the number of bits depends on what the observer already knows and what the observer is uncertain about.

In contrast, information, according to IIT, is intrinsic and relative to the system's specific state and is, therefore, observer-independent.

2. Meaningful

Information, according to Information Theory, is entirely about the statistical properties of an event (e.g. flipping over a card in a deck) and the outcome distribution (which card it is) rather than any meaning or significance the event might have.

Information, according to IIT, is intrinsic and inherently meaningful to the system itself. In fact, Tononi goes further and claims that in IIT, information is meaning.

So, our working definition is:

Information: differences that make a meaningful difference to a system from its own intrinsic perspective.

Okay, so information is specific, physical, and intrinsic (which means it is observer-independent and meaningful). We have one final characteristic to add.

4. Integration

The final piece in our definition puzzle is integration. According to IIT, it is not enough for information to be merely present; it must be integrated — the system must have cause–effect power above and beyond its parts.

So, our final definition is something like this:

Information is differences that make a meaningful difference to a system from its own intrinsic perspective, but for those differences to make a meaningful difference to the system itself, they must be integrated within that system.

That is a mouthful!

If you are wondering what integrated means and how it can explain conscious experiences… that is a fantastic question! I’ll try to answer it in our next article on IIT. But for now, I’ll leave you with something to think about.

The ins and outs of information

What if information looks different from the inside of a system than it does from the outside? As the title of this article suggests, while IIT and Shannon's Information Theory seem to offer radically different definitions of information, could they be, in fact, two perspectives of the same underlying phenomenon — the intrinsic ‘ins’ perspective and the extrinsic ‘outs’ perspective, respectively? Or are the two theories talking about two fundamentally different things that merely share the same word?

What do you think? Are we talking about two sides of the same coin, or are these fundamentally different coins altogether?

Thanks so much for reading this article.

I want to take a small moment to thank the lovely folks who have reached out to say hello and joined the conversation here on Substack.

If you'd like to do that, too, you can leave a comment, email me, or send me a direct message. I’d love to hear from you. If reaching out is not your thing, I completely understand. Of course, liking the article and subscribing to the newsletter also help the newsletter grow.

If you would like to support my work in more tangible ways, you do that in two ways:

• You can become a paid subscriber

• or you can support my coffee addiction through the “buy me a coffee” platform.

I want to personally thank those of you who have decided to financially support my work. Your support means the world to me. It's supporters like you who make my work possible. So thank you.