For eleven days my command center told me one of my products had a broken sitemap. It said so in red, on a panel I look at most mornings. The sitemap was not broken. It had never been broken. It loaded in my browser in well under a second, all 1,514 URLs of it.
It simply would not load for the agent I had running on my behalf. My site was blocking my bot. Both of them are mine.
That turned out to be a more interesting story than a bad URL, and it points directly at something every team shipping agents this year is about to walk into.
Eleven days of a red pixel
Start with my mistake, because that is the part that cost the eleven days.
The code that checks index coverage fetches a product's sitemap, then asks Google about each URL in it. The fetch was wrapped like this:
try:
urls = sorted(fetch_sitemap_urls(sitemap_url))
out['sitemapOk'] = True
except Exception:
return out # sitemap unreachable
Look at what that except clause swallows. A 404. A timeout. A DNS failure. An expired certificate. Malformed XML. A 403 from a security product. Every one of them collapses into the same result: a boolean set to false, counts at zero, and a panel rendering the words sitemap unreachable.
Unreachable was not wrong, exactly. It was useless. It described the symptom in the vaguest available terms and discarded the one piece of information that would have ended the whole investigation in about ninety seconds. The exception knew what happened. I never asked it to speak.
So the dashboard was confidently and quietly wrong for a week and a half. I had no particular reason to doubt it, because a broken sitemap on a site I had recently touched is an entirely believable thing.
The first fix had nothing to do with security. I made the error say its own name: record the exception, print it to the run log, show it on the panel. The next run answered immediately.
sitemap fetch failed for https://erateiq.com/sitemap-index.xml:
HTTPError: 403 Client Error: Forbidden
A 403. Not a missing file. Something was refusing me.
What the log actually said
A 403 from a server that hands the identical file to my browser instantly means the variable is not the file. The variable is me.
I checked from a data center instead of my desk. The sitemap returned 403. The homepage returned 403. Only robots.txt came back clean, and that one is served straight from the edge. So this was never about the sitemap. The whole site was closed to anything that looked like automation arriving from a data center.
Cloudflare's own security event log named it without ambiguity. Three requests from 20.102.103.194, at 12:03:55, 12:03:56, and 12:03:58 Central. Three, because my retry logic tries three times. The runner's address. The exact second my build failed.
Service: Bot fight mode. Action: Managed Challenge.
A managed challenge is not a block. It is a question. Answer it and you are through. A browser answers silently and the human never notices. A Python script has no idea it was asked anything at all. It sees a 403 and gives up. Which is precisely the design, and it was working exactly as intended.
Google had a key. I did not.
Here is the detail that reframed the whole thing.
While my agent was being turned away at the door for eleven days, Googlebot came and went freely the entire time. Search Console shows the site collecting impressions and clicks across the whole window. Google was crawling. Google was indexing. Google was fine.
Because Google is verified. Cloudflare recognizes Googlebot, confirms it by reverse DNS, and waves it past. My tooling is not verified and was never going to be, because I am one person with a Python script and there is no line for me to stand in.
That is not a bug either. It is the shape of the modern web. I had just never been on the wrong side of it before. There is a list. Google is on it. You are not.
The web is growing a guest list
What makes this worth writing rather than merely fixing is the timing. Cloudflare has spent the past year formalizing this exact distinction, and the free-tier defaults change in six weeks.
Automated traffic now sorts into three buckets instead of one.
| Category | What it means | Default after Sept 15 |
|---|---|---|
| Search | Indexes your content to answer questions about it later | Allowed |
| Agent | Acts in real time on behalf of a person | Blocked on ad-monetized pages |
| Training | Takes your content to train or fine-tune a model | Blocked on ad-monetized pages |
On September 15, 2026, those defaults arrive for new domains and, more consequentially, for existing free accounts that have not opted out. The site in this story is on a free plan. So are a great many small business sites whose owners have no idea any of this is scheduled.
Read the middle row again. Agent is the category for software acting on behalf of a real person, in real time. That is a customer's assistant checking your hours. That is a procurement tool pulling your spec sheet. That is the thing every vendor spent this year telling you to go build.
Meanwhile the verified lane grew a real door. Web Bot Auth lets an agent prove its identity cryptographically instead of just asserting a user agent string: HTTP message signatures under RFC 9421, an Ed25519 key per agent, a Signature-Agent header, and published keys anyone can verify. Cloudflare, OpenAI, Anthropic, Amazon, and Akamai are all shipping it. Cloudflare's verified agent category launched with roughly nineteen agents covering the large majority of identified AI browser traffic, including ChatGPT Atlas, Claude in Chrome, Perplexity's browser, and Gemini's agent mode.
Put the halves together and the picture resolves. The open web is not closing. It is growing a guest list. Signed, named agents from large operators get in. Unsigned automation gets a question it cannot answer. My sitemap fetcher is on the wrong side of that line, and so is most of the internal tooling at most companies.
The fix I did not make
The obvious fix was one toggle. Turn off Bot Fight Mode and my agent strolls in.
I want to be precise about why I did not, because the reasoning generalizes past this one incident.
Bot Fight Mode on the free plan is all or nothing. It runs outside the WAF, so you cannot write a rule that skips it for a single path. There is no exception list and no carve-out. Turning it off for a sitemap means turning it off for the login form, the API, and everything else on a production application that real customers use.
Stated plainly, the trade is: measurably less protection on a live app, in exchange for a number on a dashboard I built for myself. It is not a close call.
So I left the security posture untouched and changed my own code instead. The marketing site is an Azure Static Web App sitting behind Cloudflare, and its origin hostname serves the identical file. The tooling now reads the sitemap from the origin and skips the edge entirely. One new config value, one small change so that child sitemaps are followed on whichever host you reached the index on, and the panel went from a red error to 1,514 URLs.
No security setting was changed. The protection that was doing its job is still doing its job.
The general lesson is the one I would want a client to take: when your own automation trips your own defenses, the reflex is to lower the defense until the tool works again. That is almost always the wrong move, and there is almost always a route that leaves the wall standing.
Worth ten minutes before September 15
Three things, if you own a site.
Find out what plan you are on and whether Bot Fight Mode is switched on. Free accounts are the ones whose defaults shift in September, and they have the fewest knobs available to soften the landing.
Inventory everything that fetches your own site from outside. Uptime checks, sitemap readers, SEO tools, link checkers, screenshot services, the small script somebody wrote three years ago that emails a report every Monday. Each one is unsigned automation coming from a data center. Each one is a candidate to fail quietly, and you will believe the failure.
Then decide what you want from the agent category deliberately. Allowing search while refusing training is a defensible stance, and it is roughly what my own robots.txt already declares. Agents are the genuinely new question. If you sell anything a person might ask an assistant to look up for them, blocking agents is a decision with revenue attached to it, and it gets made for you by default if you do nothing.
Make your failures say their names
I fixed two things here, and only one of them was the block.
The block was correct behavior. Cloudflare did what I asked it to do, to a bot that could not prove it was mine. The eleven days were not Cloudflare's doing. They came from an except clause I wrote that flattened every possible cause into a single word.
I have argued before, in Work Back from the Future State, that the durable work is not the model but the harness around it, and the verification that tells you an answer is actually right. This is the unglamorous half of that argument. An agent that cannot tell you why it failed is an agent you cannot trust when it tells you everything is fine. That matters more every month, because as I wrote about Opus 5, these things increasingly run unattended in a loop while you are asleep.
Mine showed me a wall of green and one red square for eleven days. The red square was the honest one. It just was not specific enough to act on.
Make your failures say their own names. Then go find out whether the web still lets your robots in, because the answer changed while you were not looking, and it changes again on September 15.