The Machine Ran Out of Memory It Never Used

An application on the machine that runs my agent fleet started failing to allocate memory.

Every instinct said the box was short on RAM. Every reading said it was fine. More than twenty gigabytes of physical memory free, consistently, every time I looked. The page file allocated at twenty-six gigabytes with a current usage of zero. Nothing was paging. Nothing looked stressed.

And the allocations kept failing.

The number that explained it is not the one anybody checks. Once I found it, it turned out that thirty-nine percent of what each agent cost was a capability it had never used once.

The number that is not RAM

Windows tracks two different things, and only one of them is what people mean when they say memory.

Physical memory is what is actually resident. Real pages in real silicon. That is what task manager shows you, and it is where every check-your-memory instinct goes looking.

Commit charge is the other one. It is the total memory the system has promised to processes, whether or not they have ever touched a byte of it. When a process reserves a region, Windows guarantees it can back that region, in RAM or in the page file, if the process ever writes there. So it keeps a running total, and there is a hard ceiling on it called the commit limit, roughly physical memory plus the page file.

Hit that ceiling and allocations start failing. It does not matter that gigabytes of physical memory are free. It does not matter that nothing is paging. You have not run out of memory in use. You have run out of memory promised.

That is why every obvious diagnostic came back clean. Free RAM looked healthy because the fleet is not resident-heavy. Paging was zero because nothing had been forced out. The machine was suffocating on a quantity that neither reading reports.

Measured on this box right now, the agent processes commit about a third more than they ever make resident. That gap is the whole problem, and it is invisible to the reading everyone takes.

Thirty-nine percent was a browser nobody opened

So I stopped guessing and measured what a session actually costs. Across fifteen live sessions:

Thirty-nine percent of every session was two browser automation plugins.

Fourteen of the fifteen sessions had never called them. Not once. They were enabled globally, so every session started them, held them, and paid for them for as long as it lived, whether or not it would ever drive a browser.

When a number is this convenient it is worth saying how it was attributed, because a figure inferred from a difference is not really a measurement. Every process on the machine was walked, private bytes read from the operating system, and each process attributed to its owning agent by walking parent process ids up the tree until reaching one that held a session registration. Then the command line of every node process was matched against the two plugin packages. Fifteen sessions carried exactly five such processes each, seventy-five in total: a launcher and a server per plugin, plus a telemetry child.

The expensive part is the default

I enabled those plugins once, globally, months ago, for a task I no longer remember. It was the right call at the time.

That one choice was then multiplied by every session I would ever start, for as long as the setting stood. Nothing ever told me. There is no line item for it. No warning fires when a capability you switched on once gets loaded for the four hundredth time by something that will never call it.

That is the shape of the thing. A default is not a one-time cost. It is a recurring cost with no invoice. And it scales with the number of sessions you run, which means it grows in exact proportion to the thing you were excited about.

I wrote in A Name, a Lane, and a Kill Switch that once you are running more than one agent, the interesting engineering stops being prompts and becomes operations. This is that same argument one level down. Knowing which agents are running is not enough. You have to know what each one is carrying.

About seven gigabytes, and nothing restarted

Turning the plugins off did not require restarting a single session or losing any context.

Commit headroom before: 4.46 GB. After: 11.38 GB. About seven gigabytes came back as fifty-six plugin servers and eight orphaned automation browsers exited.

I want to be careful about the precision there. Commit charge moves on its own, and that delta spans a few seconds in which the machine was also doing other things. Sixty-four processes exiting dominates it, but the honest number is about seven gigabytes, not seven point zero eight five.

What one reading is worth

Here is the part I would have gotten wrong by trusting a single measurement.

While I was investigating, headroom readings taken within about an hour of each other ranged from 2.5 GB to 13.3 GB. The commit limit itself, which I had been treating as a fixed property of the machine, drifted more than two gigabytes in one evening with nobody touching it, because Windows quietly resizes the page file underneath you.

So I am not going to tell you the box was at ninety-six percent of a ninety-two and a half gigabyte limit, even though I have a reading that says exactly that. It was one sample of a moving quantity. Quoting it as the state of the machine would imply a stability that was never there.

What survives every reading I have is this: headroom fell as low as two and a half gigabytes against a limit near ninety, and the limit moved on its own while I watched. Both are true of the whole evening. The precise pairing is one photograph of a river.

What the cold start proved

Then the power went out.

Everything came back cold, which is the only honest test of a configuration change. Nobody intervened. Nobody re-applied anything.

Fifteen sessions returned at roughly 620 MB each, against 1,248 MB before. Commit headroom came back at 55 GB, against single digits the night before.

The change proved itself on the first cold start with nobody doing anything, and that is the only kind of proof worth having for infrastructure. A fix you have to remember is not a fix.

The cost of a fleet is the sum of its idle

Three things I am taking from this.

The resource that runs out is rarely the one your monitoring watches. Physical memory was fine the whole way through. Paging was zero the whole way through. Both readings were accurate and both were irrelevant, and an accurate irrelevant reading is more dangerous than no reading at all, because it ends the investigation early.

The cost of an agent fleet is not the sum of its work. It is the sum of its idle. Fourteen sessions sitting there doing nothing were each holding half a gigabyte of browser automation they would never touch. Idle agents are not free, and at fleet scale, idle is most of what you are paying for.

And the expensive part is the default. Every global setting is a bet that every future session should pay for it. Switched on once, charged forever, itemized nowhere.

Everyone is writing about what tokens cost. Almost nobody is writing about what these things cost the machine they run on. If you are running more than a couple, go and look at your commit charge rather than your RAM. The page file on this box is allocated at twenty-six gigabytes and its current usage is zero. Nothing is paging. That is not evidence the machine is healthy. It is evidence that the ceiling you are about to hit is built out of promises.

This is a note about how I work, not advice about how you should. I build software at Revelations Technology and write these as I go.