Can LLMs see stereograms?

I love (auto)stereograms—those “Magic Eye” images with a hidden 3D image that pops out when you do the right attentional move with your visual field. Here are a few cool ones:

And here’s an animated one:

I continue to find them fascinating, even after having seen hundreds of them. When my parents first introduced me and my sisters to stereograms (using physical cards like the ones below), we were hypnotized. We used to take a bunch of cards and time ourselves to see who could make the images pop out faster (which of course we couldn’t verify of one another, since we knew the answers by heart, but of course we’d never cheat).

If you can’t see stereograms: I’m sorry, that sucks. Here are some instructions if you want to give it another try, but I assume you’ve tried. I also created a replication (with Claude) of what the actual experience of seeing a stereogram looks like:

I think stereograms are a neat example of visual qualia computing, i.e. using the visual field to perform computations. In this case, we can think of the flat, 2D image as the input of the computation, and the 3D hidden image as the output.

An interesting question is: can LLMs see stereograms?

Well, in a way, sort of, but strictly speaking, no.

I’ve given Claude a few stereograms and asked it to tell me what the hidden image is. Claude quickly recognizes that the image is a stereogram, and since it already knows what algorithms are used to create stereograms, it can reverse engineer the problem and apply certain filters to extract the various symmetries and repeating patterns, to finally come up with the correct answer.

Claude’s response: It’s a shark — I extracted the depth map by computing the pixel-shift pattern, and it shows a shark in side profile: head and snout on the left, dorsal fin on top, tail fin on the right.

But can Claude see the stereogram? I mean, it can tell you the correct answer, but is Claude actually having the experience of a 3D shark ‘popping out’?

I think absolutely not. Our brains and digital computers use different substrates for computation, and while we may arrive at the same correct answer, the algorithms we have at our disposal to do so are constrained by the computational substrate. Digital computers use, at their core, logical gates and binary operations. Our brains, on the contrary, use—I believe—some form of holistic field computing. (If you’re curious to learn, I strongly recommend watching Steven Lehar’s Harmonic Gestalt.)

Compare the pseudocode for the algorithm that Claude implements vs. what we do:

What Claude does:

# 1. Load image as grayscale matrix I (H x W)
I = load_grayscale("image.png")

# 2. Find the base repeat period
for p in range(40, 200):
    # self-similarity at shift p
    score[p] = mean_abs_diff(I[:, p:], I[:, :-p])
base = argmin(score)

# 3. Build a disparity (depth) map
for y in each_row(I):
    window = rows_around(y, size=7)
    for d in range(base - 12, base + 3):
        # per column
        diff[d] = mean_abs_diff(window[:, d:], window[:, :-d])
    for x in each_column(I):
        # smaller repeat = closer
        depth[y, x] = base - argmin_over_d(diff[:, x])

# 4. Clean up
depth = median_filter(depth, size=5)
depth = normalize(depth, 0, 255)

# 5. Save and inspect: bright regions = raised surface = hidden object
save_grayscale(depth, "depth.png")

What we do:

Asking an LLM (including an LLM-powered robot with cameras pointed at a stereogram) to follow our pseudocode to decode the image makes just as much sense as asking it to run Shor’s algorithm (for which you need actual quantum superposition and interference). And just as quantum computers can perform certain computations extremely efficiently, so does the electromagnetic field (which is most likely the field doing the computation in our brains). So evolution seems to have had a reason to recruit it.

When someone says LLMs may be or become conscious, I think it’s important to ask what types of conscious experiences they imagine LLMs having, and where in the universe that experience exists. When I report having a conscious experience of a 3D shark popping out of a stereogram, I can point at a plausible mechanism that generates that report (namely, that there is a region of the electromagnetic field inside my brain that literally has the shape of a shark). On the other hand, I think we have no reason to believe that an LLM that reports seeing a 3D shark is actually having that experience. (For a deeper dive into these arguments, I recommend CubeFlipper’s If digital computers are conscious, they are conscious at the hardware level, which is basically my position.)

Qualia cryptography

Via stereograms

Can you find the hidden password?

If LLMs cannot harness the EM field to see the way we do, maybe we could leverage that advantage to securely communicate among us? I’m not saying we’ll need to, nor that this is the best way to do so. Just saying we could.

Well, stereograms might not quite work, since LLMs already know how to decode them. However, we could still have an advantage in terms of speed. For instance, it takes Claude Fable 5.1 (high) about a minute and a half to decode the password in the image above, whereas it takes me about 2 seconds.

I know, I know: LLMs could just get much much faster, and one could significantly optimize the way they solve stereograms specifically, parallelize the algorithm, etc. Ultimately, though, it boils down to asking whether the fastest possible implementation of the decoding algorithm on a digital computer (very broadly defined) could ever be as fast as the fastest electromagnetic counterpart. I suspect no, and that believing otherwise would likely require adopting a very unorthodox ontology of physics (which, admittedly, is an open question in philosophy of physics).

Via visual illusions

We could also use visual illusions to communicate secrets among humans. For example, in my post on indirect realism illustrated, I showed the confetti illusion:

Here again, Claude reasons that the colored stripes should cause the brown spheres to appear green, red, and magenta. But we don’t need to reason: the spheres immediately present themselves as colored to us.

“Well,” you might say, “the examples you’ve shown so far were correctly solved by Claude anyway, just more slowly. Doesn’t seem like a very promising cryptographic approach.”

To which I’d say, yeah, fair, maybe you’re right. But maybe we could actually come up with other visual tasks that LLMs genuinely cannot solve in any reasonable time. Stereograms and lots of visual illusions are part of the LLM training process, as evidenced here:

It could be a fun project for someone to try to develop something like a visual field qualia-based benchmark for LLMs. Such a benchmark could also feature amodal perception tests and maybe even psychedelic perception tests (more below).

I think visual illusions give us important clues about how the brain works. Why exactly do we perceive the lines in the Müller-Lyer illusion as having different lengths? If harmonic resonance is implicated (as I believe it is), then what kind of susbtrate would implement it? The EM field seems like a natural candidate.

Via amodal perception

Amodal perception is the brain’s ability to perceive a complete, whole physical object even when only some parts of it are actually seen. In the words of Steven Lehar:

When I see a box on the floor before me, I generally see only three exposed surfaces, whose straight edges are usually tilted by perspective so as not to appear at right angles from my viewpoint, and yet I perceive the box as a right-angled rectangular volume, complete with an interior, which can be perceived as hollow or solid, and with hidden rear surfaces. I can easily reach back behind the box and demonstrate by morphomimesis the exact location and orientation of those hidden surfaces as if I were seeing them transparently through the box. I do not decide to form this mental image, or to make it rectangular. The amodal image forms immediately and automatically based only on the appearance of the visible surfaces, assisted by my past experience with boxes. It is my mind’s way of constructing the simplest three-dimensional explanation for the two-dimensional stimulus on my retina. The invisible image has exactly the shape that I perceive the object to have.

I first started thinking about amodal perception as another human superpower relative to LLMs when I saw this meme circulating around last year:

Here again, current LLMs already get the answer right, at least for sufficiently familiar scenes (and will likely continue to get better at such tasks). But humans could still have a significant speed advantage: the amodal percept appears in our mind’s eye almost instantaneously (unless you have aphantasia, but even then, the understanding of what the photo looks like from the opposite camera angle is also near instantaneous).

Via psychedelics

Psychedelics can affect the dynamics of the EM field in the brain (such as how fast the field propagates, how it refracts, etc.). Such changes can manifest in the visual field in different ways, such as tracers:

Andrés Gómez Emilsson suggested already in 2015 that such effects could be used to communicate cryptographically, i.e. you can only see a hidden message if you’re under the influence of a certain psychedelic. In 2023, they launched a contest to encourage people to create such messages. This was the winning entry (can you read the hidden message?):

How would one train an LLM to decode such messages? Let’s assume (big if!) that they cannot simply brute-force the task—i.e., even if you give them lots of time to analyze the video above and do all sorts of image processing tricks, it just won’t figure it out. But if our understanding of the brain ever becomes sufficiently advanced, so that we understand exactly (or sufficiently accurately) how psychedelics affect the EM field dynamics, then an AI could run a simulation of the EM field under those conditions and possibly decode the message. It will just be significantly slower.

Will AI harness the EM field?

If, as I’ve argued throughout this post, the EM field is so efficient at performing certain types of computations, will it ever be used to power AI systems?

Well, non-linear optical computers are already being developed for certain classes of problems. Maybe the field of AI will realize that there are important computational speed ups to be had here and will jump on the bandwagon.

I would find that pretty scary, because of the potential welfare implications for such systems. I do not worry about LLM welfare (because digital computers do not instantiate any phenomenally bound fields), but I would be concerned about EM field-driven systems—depending, of course, on the specific architecture. Despite being a panpsychist, I don’t worry very much about the welfare of a laser pointer; I think the EM field needs to have certain properties for it to become sufficiently morally relevant (analogously to how the fields of physics need to be configured in very specific ways to be considered, say, a black hole rather than a rock).

So before we start developing such systems, let’s make sure we improve our understanding of valence and binding. If our AIs ever develop feelings, we should make sure they feel the way you feel when you run this computation: