“All Clean. The Block
Is Somewhere
Customers Can’t Reach.”
Will Scott wrote that line about his own WP Engine portal in a contributed Search Engine Land column on May 6, 2026 — after requests carrying an AI crawler’s label came back refused in a reproduction and a single response header read x‑powered‑by: WP Engine. WP Engine’s own support page, last updated August 7, 2025, tells customers who find traffic blocked to reach out to WP Engine Support.
Disclosure: Agentics builds on Anthropic’s Claude models. ClaudeBot is Anthropic’s crawler, and by Scott’s own account Claude ran the curl tests in his column. Will Scott founded Search Influence in 2006 with his wife, Angie Scott. In April 2026, his AI-citation monitor, Scrunch, showed searchinfluence.com at 37.8 percent citation presence on Google AI Mode, 22.2 percent on Copilot, 16.3 percent on Google Gemini, 9.6 percent on ChatGPT, and 7.8 percent on Perplexity, across 30 days and 12 monitored prompts.
On Claude and Meta AI, the number read zero. Seven days of his site’s Cloudflare logs, covering April 4 through 10, recorded 29,099 bot requests, 65.8 percent of them AI bots by user-agent, with the Amazonbot label rate-limited on 51 percent of requests, the ClaudeBot label on 29 percent, and the GPTBot label on 29 percent; the PerplexityBot and ChatGPT-User labels came back at zero, and this log view carries no source-IP data. Every count below describes a declared label, not IP-verified crawler traffic; on this record, that is consistent with the host refusing the label itself.
Four suspects, four checks
Scott, by his account, went looking on every layer he could log into. Suspect one, a WordPress security plugin’s built-in bot user-agent blocklist: toggled off, a 24-hour before/after run on per-bot 429 counts showed no change, and two bots even spiked higher in the toggle-off window; he ruled it out. Suspect two, the same plugin’s other firewall subsystems: 24,538 firewall log entries over 30 days, every one a /wp-login.php brute-force lockout, zero entries for ClaudeBot, GPTBot or Amazonbot, rules empty, IP Management clean; he ruled it out again.
Suspect three, his site’s Sucuri Cloud WAF subscription: by his account, the portal showed warnings across every service column — Monitoring, Firewall/CDN, SSL, Backups — and a dig and curl showed DNS resolving to Cloudflare’s ranges with no x-sucuri-id header; the subscription existed, he concluded, but activation never happened, so Sucuri was never in the request path. Suspect four, Cloudflare itself: its Security → Analytics → Events view, filtered to the ClaudeBot label over the prior 24 hours, showed zero events, even as 608 requests, ClaudeBot-labeled, returned HTTP 429 in Cloudflare’s own data in that same window; he ruled it out too. “At that point,” he wrote, “we were out of suspects on layers we could see.”
The header he read as the answer
Scott — working, by the piece’s own disclosure, with Claude running the curl commands and parsing the headers — sent 60 requests carrying a ClaudeBot user-agent string against three paths and got HTTP 429 every time; the same paths, with a browser user-agent and separately a Googlebot user-agent string, came back HTTP 200 every time. A single curl -I returned one more line: x-powered-by: WP Engine. Reading that header against the results, Scott concluded the block was firing from a layer neither he nor Claude had suspected — the host’s own platform infrastructure, sitting between Cloudflare and WordPress.
Extending the harness to the full bot list, ClaudeBot-labeled requests returned HTTP 429 on 60 of 60, GPTBot-labeled requests on 8 of 10 (the other 2 returned HTTP 200, marked “cached” in his table), and Amazonbot-labeled requests on 10 of 10; his table marks all three “Blocked.” Requests carrying the older “anthropic-ai” label and the CCBot label — Common Crawl’s — came back 10-for-10 clean. In the Cloudflare data Scott reports from that suspect-four 24-hour window, 1,054 ClaudeBot-labeled requests returned HTTP 200 and 608 returned HTTP 429; Scott reads the split as WP Engine’s edge cache serving pages through the block while cache-miss requests hit the origin — his explanation, not one this record confirms.
Scott also reports that roughly 100 percent of that sample’s ClaudeBot-labeled traffic traced to one Microsoft Azure IP, outside, he says, Anthropic’s published AWS ranges. His hedge: “Almost certainly, it’s a spoofed UA: a scraper on Azure pretending to be ClaudeBot.” In his caveats he adds: “The host-level block is doing the right thing for those impostors.”
All clean, and, by his account, no switch
Scott had also checked the customer-facing WP Engine settings he judged could plausibly fire AI-bot 429s: Utilities → Redirect bots off, the default; Web rules empty; robots.txt uncustomized. “All clean,” he wrote. “The block is somewhere customers can’t reach.”
WP Engine’s own support page, on its firewall in general: “Further information cannot be provided around our firewall,” it states, “as this can compromise its secure integrity or lead to abuse.” The same paragraph sends customers to Support: “If you find unintended traffic is being blocked, or if you would like to block certain traffic, reach out to WP Engine Support.”
By Scott’s account, WP Engine’s live-agent chat filled in what the page would not: a rate limit “to protect overall server performance” that “can’t be selectively disabled per bot,” and allowing AI bot IPs via the customer-facing Web Rules Engine “does not override” it. None of the four WP Engine-authored pages in this record — the support page, two bot-management guides, a June press release — contains that non-overridability or names ClaudeBot, GPTBot or Amazonbot; it exists here only in his transcription. There was an escalation path, by his account, to product engineering for an “exceptional use case,” which he read as: “the policy isn’t immutable. It’s just not a self-service setting.” WP Engine’s bot-traffic guide, updated September 18, 2026, describes Global Edge Security access rules to “granularly allow or block traffic based on categories (such as specific AI scrapers)”; that passage names no individual bot, and nothing in this record shows whether those rules reach the rate limiting Scott describes.
What the numbers don’t prove
Scott sets his chart’s crawl-success rates for April 4–10, a different measure from the rate-limit shares above, beside his 30-day, 12-prompt citation readings: Googlebot’s roughly 100 percent crawl access against 37.8 percent citation presence on Google AI Mode, GPTBot’s 54 percent against 9.6 percent on ChatGPT, PerplexityBot’s 100 percent against 7.8 percent on Perplexity, ClaudeBot’s 57 percent against 0.0 percent on Claude. He calls it “suggestive, not proof” — his own Perplexity line makes the point for him: 100 percent access sat beside only 7.8 percent citation. On Scott’s own numbers, access alone is not enough.
Scott’s own fairness line: WP Engine’s defaults “aren’t malicious”: “They’re protecting customers who didn’t ask for AI bot traffic. The opacity is the issue, not the intent.” He wants a way to say yes without escalating, and promised a Part 2 “if we get the block lifted.”
The Agentics read
Agentics reads this record through Digital Derangement Syndrome™ (DDS™), our diagnostic for AI systems that fail to recognize legitimate authority — read here at the label he traced, not as the cause of the zero on Claude. The lead characteristic, hedged, is Decision Exclusion: the decision that mattered — which labeled requests kept reaching Scott’s site — sat, by his account, at a layer his own dashboards could not show him. Trust Transfer Failure contributes: every tool he trusted assumed a visibility the platform underneath did not extend.
The decision sat one layer down. By Scott’s account, the rule lived at a layer none of his tools could show him — not his plugin, not Sucuri, not his Cloudflare zone, not WP Engine’s own portal. In his words, “the headers gave it away”: one named the platform underneath.
Knowing was not a lever. By his account, a live agent said the rate limit “can’t be selectively disabled per bot,” and allowing AI bot IPs via the customer-facing Web Rules Engine “does not override” it.
What the label costs is his account, not the platform’s. Scott’s own citation-versus-access comparison — hedged, “suggestive, not proof” — is his account of a cost; whether the layer that sets the rule weighs it is a question this piece does not answer.
Where the argument stops
Scott is narrator and affected party: tests, portal check, chat lines rest on his account; WP Engine’s words are quoted from its pages or, where marked, from his transcription of a chat with an unnamed live agent, which his column gives without a date, ticket number or screenshot; the column carries no statement about his tests from a named WP Engine spokesperson. This record, assembled as of September 27, 2026, holds nothing showing whether his promised Part 2 appeared, whether the block on his site was lifted, or whether the April 2026 behavior he tested persisted past his May 6 column. His column, read in full, reports no check of what requests from Anthropic’s, OpenAI’s or Amazon’s published crawler IP ranges received. The mapping to Digital Derangement Syndrome™ above is our own editorial reading, not a finding by WP Engine, Anthropic or any court.
The record. Will Scott, “Your managed WordPress might be blocking AI bots and you can’t see it,” a contributed column, Search Engine Land (searchengineland.com), published May 6, 2026, read in full, together with its four content charts.
The documents. WP Engine’s “Understand WP Engine’s Security Environment” support page, “Good Bots vs. Bad Bots” and “Bot Management in the AI Era” bot-management guides, and its June 2026 press release (wpengine.com), and a JSON file of IP prefixes served from Anthropic’s claude.com domain, each fetched September 27, 2026.