Heisenbugs at Scale

You know the bug. It happens once a week in production. It never happens in your test environment, no matter how hard you try to reproduce it. You add logging and it stops happening. You add more load and sometimes it comes back, sometimes it doesn’t. Your team spent three weeks chasing it. You never found it. You just added a retry loop and moved on.

That bug has a name. It’s a Heisenbug, and we’ve been shipping them for decades.

The name comes from Heisenberg’s uncertainty principle: the act of observing the system changes the system’s behavior. In software, it’s what happens when you try to debug non-deterministic code. The moment you add a breakpoint, the timing shifts. The race condition disappears. The bug vanishes the instant you look at it.

We did not choose this. We inherited it.

The non-determinism we already live with

Software was supposed to be deterministic. Same input, same output, every time. Call it twice with the same arguments and you get the same result. That was the promise.

Reality delivered something messier.

Floating point arithmetic is not deterministic across platforms. The order in which operations happen in a multi-threaded system is not deterministic. The timing of network packets is not deterministic. The order in which your distributed system’s nodes respond to each other is not deterministic. The state of your cache is not deterministic. The result of reading from disk is not deterministic if the disk has any say in the matter.

We built entire systems on top of this non-determinism. We had to. The alternative was to throw away the power that made those systems valuable.

So we learned to manage it.

How we learned to live with non-determinism

The first thing we did was accept that some bugs only exist in production. We called them “intermittent failures” and treated them as a category of problem distinct from regular bugs. We built tools to deal with them.

Deterministic replay. Record every message, every timing decision, every random choice. Replay it in the lab and suddenly the bug is reproducible. This works because the non-determinism is external. You didn’t understand it, but you can record it.

Idempotency. If an operation can be safely executed multiple times and produce the same result, we can retry it without worrying about the side effects. It’s a hedge against non-determinism. The operation might fail the first time, but try again and it might succeed. The system as a whole converges to the right state even though individual operations are flaky.

Chaos engineering. If we accept that some things fail non-deterministically, we can intentionally introduce failure and see how the system responds. We kill random processes, delete random packets, inject delays. We stress the system until we find the edge cases. We treat non-determinism as a property to understand and manage, not a bug to eliminate.

Observability. Since we can’t trace our way to the root cause of a non-deterministic failure, we watch for the symptom instead. We log everything, emit metrics, build dashboards. We monitor the system in production and watch for the moment when things go wrong. Then we correlate the logs and try to figure out what happened.

Immutability and pure functions. We reduced our surface area for non-determinism by building systems out of functions that have no side effects, that don’t depend on hidden state, that can be executed in any order and still produce the same result. We can’t eliminate non-determinism, but we can fence it in.

These tools worked. They were hard-won through years of pain. We learned to ship systems that were fundamentally unreliable but reliably managed their unreliability.

The bargain we struck

We accepted non-determinism on two conditions.

First, we understood the non-determinism. We knew it came from concurrency, or floating point rounding, or network timing. We could reason about the bounds. We could say “this will fail once every million times” or “this will fail under high load” or “this will fail on this hardware but not that hardware.” The non-determinism had a cause, and the cause was knowable.

Second, we could manage it. We could add a mutex to avoid the race condition. We could use fixed-point math to avoid the floating point rounding. We could add exponential backoff to handle the network timeout. The non-determinism was a problem, but it was a solvable problem.

We never loved this situation. But we understood it. And we built the systems anyway because the payoff was worth it.

Then AI arrived

Large language models are non-deterministic. Not as a side effect, not as a limitation, but as the core design.

The temperature parameter controls randomness. Lower temperature means more deterministic output. Higher temperature means more creative output. But you cannot set temperature to zero and get pure determinism. You can get closer to it, but not there. And you wouldn’t want to. At zero temperature, the model is just picking the single most likely token at each step. That produces repetitive, boring output. The magic happens in the randomness.

The training process is non-deterministic. Different random seeds during training produce different models. The model parameters drift as you train. If you retrain the same model on the same data with a different random seed, you get a different model.

The output is non-deterministic. Run the same prompt through the same model twice and you get two different responses. Sometimes they’re similar. Sometimes they’re wildly different. There’s no way to predict which.

Here’s the part that should worry you: this is not a bug. This is how it works.

We’re shipping Heisenbugs as features

A traditional Heisenbug is a failure you can’t reproduce because the act of observing changes the system. The system is deterministic underneath, but the non-determinism is so delicate that any intervention breaks it.

LLMs are different. They’re deterministic underneath (if you know the seed, you can reproduce the exact output). But we don’t ship the seed. We ship the model, which samples at every step based on the temperature parameter and the internal state. The non-determinism is intentional and necessary.

We are now taking this approach and embedding it in production systems.

A chatbot that sometimes gives good advice and sometimes doesn’t. A classifier that marks the same email as spam one day and safe the next. A code generator that works half the time. A recommendation system that suggests different things to the same user depending on when they ask.

And we’re doing this in industries where non-determinism used to be unthinkable. Medicine. Finance. Autonomous vehicles. Law. Systems where the requirement was always “this must work correctly every time.”

The old tools don’t work

Deterministic replay: you can’t replay an LLM’s decision because the decision-making process is probabilistic. Even if you knew the exact prompt and the exact model weights and the exact random seed, you’d be capturing the non-determinism, not eliminating it.

Idempotency: an LLM’s response is only idempotent if you set temperature to zero, which defeats the purpose.

Chaos engineering: how do you chaos-test a system that’s already chaotic by design? You stress-test it and sometimes it works and sometimes it doesn’t, and the gap between those two outcomes is the temperature parameter.

Observability: you can log what the model said, but you can’t log why. The decision happened inside a neural network. There is no reason to report, just a high-dimensional probability distribution and a random draw.

Immutability and pure functions: the model is not a pure function. It has hidden state in the form of the model weights. Even if you give it the same input, the weights have changed (maybe someone did a fine-tuning step). The output is different.

We don’t have a playbook for this.

What we’re doing instead

We’re pretending observability is enough.

“We’ll just monitor the model’s outputs and flag anything that looks wrong.” Except you don’t know what “wrong” looks like until after it causes damage. The model doesn’t produce an error message. It produces a confident, coherent, plausible answer. That answer might be completely wrong. You won’t know until someone notices.

We’re building feedback loops and hoping they catch the drift. “We’ll retrain the model on user feedback.” Except the model is drifting continuously. Today it’s biased one way, tomorrow it’s biased another way. The feedback loop lags behind reality.

We’re documenting the limitations and hoping that’s a defense. “The model sometimes produces incorrect outputs.” Yes, but you shipped it in production anyway. You shipped it in a system where that sometimes matters.

We’re adding ensemble methods and hoping that averaging wrong answers produces a right one. “We’ll run the prompt through five models and take the majority vote.” Sometimes that works. Sometimes all five models are wrong in the same way.

We’re crossing our fingers and saying “artificial intelligence will figure it out.” Maybe it will. Maybe it won’t. We’re betting real money and real lives on maybe.

The bug you can’t reproduce

Here’s the hardest part of Heisenbugs. They’re not actually bugs. The system is working correctly. The non-determinism is real, not a defect. There’s nothing to fix. You have to ship a system that fails sometimes and accept that as a feature.

With traditional Heisenbugs, you could eventually fix it. You added synchronization, or used a different algorithm, or changed the hardware. The non-determinism was a problem to solve.

With LLM-based systems, there is no fix. The non-determinism is the system. You can adjust the temperature. You can prompt-engineer your way toward better outputs. But you cannot make it deterministic without throwing away the whole approach.

So you accept the non-determinism and you monitor it and you hope it doesn’t matter too much.

Except in some cases, it really matters. A medical diagnosis system that gives different answers to the same patient depending on how it wakes up. A financial system that approves a loan one day and denies it the next, for the same applicant. A legal AI that interprets the same statute differently each time.

These are the Heisenbugs we’re shipping at scale. And unlike the Heisenbugs of the past, there’s no fix coming. There’s just management and monitoring and hope.

The reckoning

We learned to ship non-deterministic code. We learned it was hard. We built tools. We built practices. We accepted it as the price of power.

Then we doubled down.

We took the hardest kind of non-determinism (the kind you can’t understand) and built an entire industry around it. We embedded it in every application. We made it the default solution to problems that used to require careful, deterministic code.

Some of this will work out fine. The non-determinism won’t matter. The system will be tolerant enough that a little variance in output is not a problem.

Some of it will not. We will discover, too late, that the system was making crucial decisions in a way that depended on randomness. Someone will be harmed. We will be shocked. We will say “we didn’t know this could happen.”

We did know. We’ve been shipping Heisenbugs for decades. We know they’re hard to find, hard to fix, and hard to understand. We just decided to scale them to a new level and hope observability was enough.

Maybe it will be. Maybe we’ll build the right monitoring and catch the problems before they cascade. Maybe we’ll get lucky.

But the hardest bugs to fix are the ones that fail sometimes and succeed sometimes without a clear differentiator. We’re about to find out what happens when you build an entire stack of business logic on top of systems that are designed to do exactly that.

The Heisenbug is no longer a bug. It’s the system. And we’re shipping it everywhere.