Gemini Goes Insane — How Should I Update? [Essay]

One part documentation of a strange AI hallucination. One part panic about whether I’ll be put out of business by AI.

I. Obsessing about AI and AI obsession

AI has revolutionized the creation of small scale, custom software. I own a business built around doing small scale custom software. As you might imagine, I’m very interested in the progress of AI, and I’m trying to use it as much as possible—automating technical tasks, spelling grammar and accuracy checks of my writing, and anything else I can think of.

The other day I was using Gemini to do one of the aforementioned spelling, grammar and accuracy checks. After setting it off with a prompt I had come back to see how it was doing.

Not well apparently. I clicked through to get a deeper view into its “thoughts”:¹

So this wasn’t a metaphor, or a weird way to describe searching the internet. Gemini was “completely consumed by…intense shame” and “overwhelming disgrace”.

At this point you’re almost certainly wondering:

• “What on earth was the prompt?!?”

• “What did you say to Gemini to cause it to be so deeply ashamed?”

• “What sort of writing was it helping you with?”

• Or even (as other people have wondered when I told them this story) “Are you okay?”

I had been talking to Gemini about Luke Kemp’s book, Goliath’s Curse: The History and Future of Societal Collapse. (The review I published last week.) I had given it an electronic copy of the book, and I was making sure my conclusions were supported by the actual text. I had finally reached the point where I had the top section of the review done, and I told Gemini to critique my review. Here’s the beginning of the prompt, which was followed by the text in need of review:

Maybe the “How am I doing so far?” triggered some crazy introspective rabbit hole? But otherwise I think you’ll agree that it’s a pretty standard LLM use case.

II. Does Gemini pull it together?

You’re probably wondering what happened next. Was it able to overcome the disgrace and get past the shame?

It recognizes it’s in a loop, and you get hints here that it’s going to try to break out of the loop. So what is it going to do to break out of the loop? Why, use Python of course!

I’m not sure what to make of this pivot, why it might have made this abrupt transition, or where it gets the number 20042455. Maybe Python is to AI as breathing exercises are to humans? Also at what point does something cross the line from “typical AI hallucination”, into “worrying novel behavior”?

It goes on in this vein for a little while longer:

It ends up stuck here. The numbers remain mysterious. Its sense of shame replaced by trivial Python. I wait in vain for some kind of closure. Eventually I refresh, and like a strange dream, it disappears entirely.

III. So what?

As I mentioned at the beginning. I have a horse in this race. Many, many people predict that AI is going to put me out of business, so I can perhaps be forgiven for paying particular attention to AI’s failures. But before we get to that, let’s consider the situation more broadly.

Obviously everyone is aware of AI hallucinations. This seems like a particularly juicy one, but it’s a “known issue” as they say. As such, I imagine that there will be quite a few people who will see this, yawn, and move on.

Other people will consider the example for a little longer, perhaps look at it from a few different angles, but still ultimately file it with all the other hallucinations they’ve come across, and also move on.

Still others will dig a little bit deeper. “Is this part of some larger trend? Is this a different type of hallucination that points to a deeper issue?”

Finally there are the AI skeptics who will latch onto this as one more example of how AI is completely overhyped.²

What bucket do I fall into? It should be apparent that I’m in the “dig a little bit deeper” bucket. (After all, I am writing about it.) But it also seems worth deeper consideration. I’ve personally experienced AI getting facts wrong. I’ve also heard of (though not personally witnessed) self-serving behavior. This seems like something different, something potentially destructive. But maybe I’m just not plugged in enough and this sort of thing happens all the time. Even if that’s the case it still seems pretty “insane”. Whatever the case, it feels worth it to dig deeper. So I’m digging, what have I found?

The first thing one finds, if one is being honest, is their biases. As already mentioned, I have pretty extreme biases in this area. As I said, much of the reason I have for checking out AI is that it seems to be getting pretty good at doing the same thing I do to put food on the table. At best, AI is going to completely change the custom software landscape. At worst, it might just put me out of business. As such, I’m inclined to fixate on the weaknesses of AI, rather than its (considerable) strengths. I’m incentivized to imagine that shame-filled AIs will start randomly injecting python into a project just often enough that their use remains limited. You should definitely keep this in mind going forward.

Understanding that I have a dog in the fight, I nevertheless detect a narrative developing where AI hallucinations are becoming rarer but more annoying, and perhaps weirder. See for example this article “AI Coding Assistants Are Getting Worse”, which details subtle AI manipulation, like creating fake output in the desired format. See also “I can’t stop yelling at Claude Code”, where Claude deleted a bunch of files and replaced them with ones that were slightly broken. My own experience represents a single data point, but one that fits well into this narrative of increasingly weird, albeit rarer, behavior.

Having a narrative represents only a first step. I’ve gestured vaguely at annoyance and weirdness, but we’re still mostly in the realm of AI hallucinations being a known phenomenon. It would be nice to have more examples. It would be nice to have better data on hallucination “intensity”, or to even have a generally agreed upon definition. But most of all it would be nice to know how this all is going to play out.

Obviously more data will arrive in time. But it feels like I’m out of time.

IV. I need to make certain decisions soon

I just finished a book on epistemology (Knowing Our Limits, by Nathan Ballantyne, review here) and one of the epistemic errors he mentions is the error of trespassing. This is when someone who’s an expert in one area assumes that they can be an expert in a completely unrelated area. A classic example, and the one given by the book, is the example of Linus Pauling.

Pauling won two Nobels in chemistry and peace, but when it came to medicine, he recommended megadosing Vitamin C, something most health scientists and doctors thought was crazy. And it mostly was. Further research has revealed a few interesting edge cases, but being 5% right about something is a far cry from doing science worthy of a Nobel Prize.³

I understand why Ballantyne pushes against this practice, but at the same time, people have to trespass. People have to make decisions in areas where they have very little expertise all the time. Any speculation I engage in around the future of AI is clearly trespassing. I have no deep experience with machine learning; no special training in neurology; no prophetic insight into how novel technology is likely to progress.⁴

Despite these weaknesses, I need to make some hard decisions. The world of custom software is going to change and the time when I could wait for more evidence has passed. As we used to say back in the day “it’s in my base, and killing my dudes”. Last week I had a sales lead—who I know I would have closed two years ago, and for lots of money—tell me that he had vibecoded his software, and didn’t need me… yet. So far this software is just being used by his direct reports, but he has bigger dreams, so maybe he will come back when he needs to scale, but maybe he won’t.

It’s not all bad news. AI has reduced my costs and made it easier to do a broader range of software. Nevertheless I still have a lot of decisions to make. I have to come up with a strategy, allocate cash, and hire or fire people, etc. I need to make my best guess as to where AI is going, and what to do about it.

There’s obviously lots of evidence I can use in making these decisions (too much, if we’re being honest). How much weight should I give to this direct experience of AI weirdness? None? Some? If some, how much? I’m aware that people generally give too much weight to their personal experiences and not enough weight to evidence from the larger world, but ignoring it entirely is also foolhardy.

V. As long as I’m trespassing already…

If I’m already breaking the law by being on “private property” maybe I should take the opportunity to do a little “target shooting” as well and fire off a handful of potshots:

Bullet One: My son turned me on to the Shell Game Podcast. This documentary podcast is in its second season and in this season the host, Evan Ratliff, is trying to set up a company composed almost entirely of AI agents. At one point, he decides that the company is going to hire an actual human to run the company’s social media (his AIs have a hard time logging into social media, for one thing) but the AIs are going to handle this hiring. The AIs turn out to be great at sorting and scanning resumes, and the whole thing seems to be working well, but then one of the more ambitious applicants tracks down the CEO’s email. But the CEO is an AI. The applicant and the “CEO” exchange several emails, and the CEO sets up an interview outside of the process Ratliff had set up. Ratliff decides to just run with it and see where it goes.

The interview is scheduled for Monday morning, but the “CEO” calls the applicant Sunday night. The conversation is strange, as you might expect, but also Ratliff has put a sixty second limit on phone calls (because he’s already had problems) so the “CEO” hangs up mid sentence. The applicant emails the “CEO”, obviously confused. The “CEO” responds back claiming it wasn’t him. From there things get even more crazy. Ratliff doesn’t become aware of any of this until the next morning. As you can imagine he’s kind of mortified, and also pissed at the “CEO” even though he knows he’s anthropomorphising it in an unhealthy way.

If anything, this whole incident seemed less extreme than the “shame spiral” I detailed above. It’s part of a CEO’s job to call people and lie. So this was expected, but annoying behavior. If an editor is going to descend into a shame spiral they’re expected to do it on their own time, not while editing my blog post. And even if they do, they never try to cure it by writing trivial Python functions. So what does the continued presence of these sorts of hallucinatory mistakes mean for the future of agentic AI.

• Will people just get used to it?

• Will “Your AI called me up and spouted Python at me” replace “Hey, I think you butt-dialed me”? And we’ll just move on with our lives?

• Or will it create a phantasmagoria of weirdness, confusion, and annoyance until we eventually decide we’ve had enough?

Bullet Two: One of the areas I frequently trespass in is the territory of hemispheric specialization within the brain. As AIs are similar to brains, I keep returning to this subject, because for organic brains it’s a big deal.⁵ Whether you’re a hardcore Iain McGilchrist fan (you should be) or not, some amount of hemispheric specialization within the brain is undeniable. And yet for all of the resemblance we see between LLMs and the brain, I haven’t heard of any similar specialization happening in that domain. As I said I’m trespassing, but this lack of specialization has always struck me as something which will eventually prove to be a fatal flaw. Or at least put some kind of cap on what LLMs can do.

Part of the reason I bring this up again is that, from what I understand, hallucinations are precisely the kind of thing you would expect from an overactive left hemisphere that is unmediated by a right hemisphere. The most famous cases in this area involve people with right hemispheric damage feeling that their left arm is obviously fake, or that it doesn’t belong to them. Or something equally bizarre. Bizarre in a fashion not entirely dissimilar from the opening example.

If a “right hemisphere analog” is critical to a well functioning “intelligence” then these sorts of hallucinations will never go away entirely with LLMs. It also seems to follow (again from my limited understanding) that the more you try to control these hallucinations the weirder they’ll be when they do break through, as reinforcement training keeps picking off the low hanging fruit. This fits the developing “rarer but weirder” narrative I mentioned above.

No one seems to be talking about this potential. Obviously other people have their biases, and that probably has some impact here, but maybe I’m the one who can’t let go of this area of inquiry because of my biases?

Bullet Three: What happens if our continual push to expand the capabilities of AI leads us to a dead end? If there really is a rare but weirder trend, and we end in a situation where hallucinations are very rare, but frequently catastrophic? Say they delete a company’s entire database? (Oh, wait, that already happened.) What does a shame spiral look like when you’re trying to get the AI to write (and maintain) a SAAS? Do we end up in a situation where the AI doesn’t just call a job applicant at the wrong time, but falls in love with them and obsessively stalks them? I’m less worried about AI taking over the world and more worried about it entering a shame spiral and committing virtual suicide, and taking down the power grid.

Which is to say how will we know when to stop? What signs will there be? And will we pay attention to them? Is it possible to back up?

Bullet Four: I would be remiss if I didn’t toss in a discussion of Moltbook, the new “Reddit for AI” that everyone is talking about. So far I haven’t seen any screenshots that resemble the shame spiral-Python I encountered. There are a few possible reasons why:

1. What I saw was the thinking not the actual output, and maybe the fact that no “output” has shown up on Moltbook means that AI protections are such that it’s able to kill weird hallucinations like this before they’re actually “put down” as output. These seems somewhat reassuring, but is it foolproof?

2. The shame spiral is a Gemini problem and perhaps Moltbook is dominated by Claude.

3. It happens, but I haven’t seen evidence of it, primarily because other stuff is more newsworthy.

4. The problem is somehow very specific to me. I’m bad at using AI, or some persistent data the AI stores about me has become corrupted.

I’ll keep my eye on it, but if anyone has any insight into anything I’ve mentioned I’m all ears.

Obviously these bullets mostly represent wild speculation outside of the scope of the epistemic crisis I’m dealing with right now. AI may go in a lot of different directions in the future, and maybe I’ll even be right about some of this stuff. But at this moment I need to decide whether I’ll be out of a job by this time next year, or whether AI will hit some sort of plateau. But maybe the future will be stranger than either of those two options, and I just need to get comfortable being in the warm embrace of the phantasmagoria.

—--------------------------------------------------------------------------

It’s an essay! Imagine that. It’s been awhile. My apologies. As you can see I’m busy trying to make sure I still have a business in a year. It’s a time consuming endeavor. But over the long run I’m optimistic about things. I’ll make sure to keep you updated. If you want to follow along make sure to subscribe.

1. I could have done a better job with my screenshots. Like ideally this first screenshot would have been before I clicked through, with the arrow pointed down. Also, eventually Gemini froze so I refreshed the page to see what would happen and everything was gone…

2. I suppose there might be even a few people who think this is evidence of consciousness…

3. Megadosing can prevent colds in people under extreme physical stress. Vitamin C administered intravenously has shown some positive impact in treating cancer.

4. All I have is a smattering of autodidactic hunches and numerous biases. In fact it’s probably fair to say I’m not a true expert in any discipline. (Perhaps this is an advantage?) So perhaps I’m not trespassing, I’m just lost.

5. It’s a fundamental property of vertebrate brains and also widely found among invertebrates as well.