Why Kubernetes keeps killing your AI agents

Your agent tier is sitting at 12% CPU, so Kubernetes does the sensible thing and scales it down.

It also takes forty in-flight tasks with it.

Nobody misconfigured anything. Every pod was sized correctly, and every control loop did exactly what it was told.

That is a normal first week running AI agents on a cluster built for something else.

By the end of this post you should be able to tell which of four very different problems you have, and whether yours is an afternoon of config or a real architecture change.

Kubernetes could schedule those workloads fine; what it could not do was see them.

Kubernetes made three assumptions, and your agent breaks all of them

Think about a normal web service. It answers a request in milliseconds, holds nothing precious in memory, and if it dies you start another and nobody notices. Kubernetes was built for that. It made three bets: pack small interchangeable units onto machines, scale them on CPU and memory, and kill one freely and start it elsewhere. Those bets were right, and they are much of why Kubernetes won.

Now picture an agent. It spends most of its life waiting on a model API, using almost no CPU while doing the most expensive work in the pipeline. Then a big tool result lands and its memory balloons for a few seconds. And it is not interchangeable, since the conversation and the working files live in that one sandbox. Kill it and you have not restarted a service. You have lost somebody's session.

None of that is a design mistake. It is a good design running outside the world it was built for.

The Kubernetes project agrees, which is handy to know before you argue with your platform team. Its own post, Running Agents on Kubernetes with Agent Sandbox, calls agents isolated, stateful, one-of-a-kind workloads that need a new abstraction. So the mismatch is settled. Your question is different. Can you fix this on the cluster you run today, or do you need something else underneath?

Agent memory is not a bell curve, and that breaks your autoscaler

One task uses 500 tokens. The next uses 500,000. Memory follows the same curve, so "average memory use" tells you nothing about what the pod needs in ten minutes.

ScaleOps reproduced what happens next. A pod with a 2 GiB memory limit meets a task that wants 3 GB, and the kernel kills it mid-task. Memory is the one resource with no forgiveness. If a pod wants too much CPU, Kubernetes slows it down. There is no slower version of needing more RAM than you were given. The pod dies.

So you do the defensive thing and ask for the worst case, not the typical one.

But there is a catch. A Kubernetes autoscaler does not measure how busy a pod is in plain terms. It measures what the pod uses as a fraction of what it asked for. Ask for many times what you usually need, and a slammed pod still reports low usage, forever. You bought safety from being killed and paid by blinding your scaler the other way.

Starting a pod used to be cheap compared to how long it lived

A web pod takes a second to start and then serves traffic for days, so nobody minds the second. An agent might exist for the length of one tool call, with a human watching. Suddenly that second is most of the job. And "agent cold start" names at least four different measurements, so published numbers sit miles apart.

At the fast end, you grab a container that was already running. At the slow end, a cloud provider finds you a new machine, pulls a huge image onto it, and loads model weights into memory. Both get called "cold start". If someone quotes you a start-up time, ask which one they did.

Then the part you cannot tune your way out of.

Kubernetes scaling is a chain of pollers. Something scrapes metrics every so often. KEDA checks those metrics on its own timer. The autoscaler syncs on another. And before scaling down, it waits out a stabilization window, so your cluster does not thrash on every blip.

Add those up and the loop reacts in minutes, while agent bursts last seconds. It is a smoke alarm that checks the room every five minutes, which is fine for a slow smoulder and useless for a flash fire. No setting fixes it, since the delay is the design.

Which is why everyone reaches for warm pools. The honest description of one is that you pay to keep capacity doing nothing, so somebody else does not have to wait. Twenty hot sandboxes is twenty idle containers on the bill. That can be the right trade, but it is a trade rather than a fix.

Two projects are rethinking this, and they are not the same project

Most coverage blurs them together, so let me separate them.

Agent Sandbox keeps the model you know: one pod per agent, on the normal scheduler, with new resource types for sandboxes, templates, claims and warm pools. It is deliberately modest, and it does not isolate anything itself. It hands that job to a stricter container runtime you pick per workload. If you read an older write-up calling it experimental, note that it has shipped steadily since August 2026, though its API is still beta.

Agent Substrate answers a different question. It keeps Kubernetes for managing machines and long-lived worker pods, then packs many agents onto far fewer of those workers, freezing an idle agent to storage and handing the worker to somebody else. Think hot-desking rather than giving everyone an office. Tim Hockin, one of the original creators of Kubernetes, co-announced it, and the doc is blunt about why. A couple of seconds to start a pod is wonderful for something that runs for hours, absurd for something that runs for milliseconds. In kagent's version the security boundary wraps the agent, not the worker pod.

I am not recommending it yet. The doc says at the top that much of this is aspirational, so its headline figures are targets rather than measurements, and it is a young project with a big pile of open issues.

When the code is written at runtime, the container is not enough

For a normal service, a container is a fine boundary. You wrote the code, so you know what it does. An agent is different. Its code is written at runtime by a model that just read something a stranger sent it. And containers, for all the isolation talk, still share the host machine's kernel. It is less "separate houses" and more "separate rooms in one house", with thin walls and the same foundation.

The fix is a thicker wall. Pick a runtime that puts its own kernel between the workload and the host, chosen per workload with RuntimeClass.

The industry has noticed. The CNCF's Kubernetes AI Conformance now includes a sandboxing requirement, explicitly because AI agents may run untrusted code. Read the small print, though, since it is a should, not a must.

Now the deflating bit. Strong isolation limits the damage a compromised agent can do on the box. It does nothing to stop the agent being talked into misbehaving. The paper Containing the Autonomous Operator takes the right posture: assume the attacker already has the agent, and design from there.

What actually keeps your data from walking out the door sits at the network edge:

• Block outbound traffic by default. Default-deny egress means the agent reaches only what you allowed, not the whole internet.
• Block the cloud metadata endpoint. That internal address is how a workload asks the cloud provider for credentials. An agent has no business asking.
• Never let the agent hold the keys. Add credentials at a proxy on the way out, so the secret lives on the path and not in the sandbox.
• Make sandboxes single-use. One per session, thrown away after. A sandbox that survives across users leaks one customer's files to the next.

A perfectly isolated micro-VM with open outbound internet is an exfiltration channel with extra steps.

Most teams do not have the problem the fancy architecture solves

Substrate is built for millions of agents. If you run fifty, that is not your architecture, and reading about it will mostly make you anxious. The strongest production agent talks of 2026 reportedly added no new infrastructure, coordinating everything with the Kubernetes parts already in the cluster. And the horror story told on those stages, an agent that logged into production and rebooted servers, happened because it had the credentials. That is a permissions bug in a scheduling costume.

Bring the same scepticism to the economics. The headline argument for rebuilding everything is that agents sit idle 90% or more of the time. That number comes from the application its backers filed to get the architecture adopted, with no method and no sample behind it. I went looking for an independent measure and found none.

What does exist is a measurement one layer down, on the GPUs:

Three quarters of the bill buying nothing is a real and expensive finding. It is just not evidence about the agent processes, though people quote it as though it were.

Which of the four problems do you actually have?

Nearly everything here is one of four problems wearing the same symptoms. Naming yours is most of the work.

• Your autoscaler reads the wrong signal. Scale on queue depth or token throughput rather than CPU, shorten the scale-down window, and stop padding memory requests to dodge being killed. This is configuration.
• Your agent holds credentials and network access it should not. Lock down egress, give each agent its own identity instead of a shared token, add a credential proxy, tighten permissions. This is policy.
• You run model-written code against a shared kernel. Pick a stricter runtime for those pods. Still Kubernetes, different runtime underneath.
• You really do have thousands of mostly-idle agents. Or you must start one per tool call while a human waits. Warm pools first, and only at the far end something like Substrate. This is architecture.

Only the last means leaving the one-pod-per-agent model behind. The other three are a Tuesday afternoon.

What to do this week

Start with one question. What does your agent tier scale on? If the answer is CPU, that fact probably explains most of what you are seeing, and swapping it for queue depth is an afternoon with KEDA.

Then check two things in your manifests. Did anyone pad memory requests to stop pods being killed? And do your sandbox pods name a runtime class? Write down which of the four problems you have before anyone proposes a new substrate. It is a short note, and it will save you a long quarter.

Rather than being replaced, Kubernetes is being re-layered here, as it was when it learned to run databases. What is dying is the reflex of describing every workload as a pod. The software still does exactly what it is told. We just stopped being the ones telling it.