Agent-invisible, defined and measured

A site is agent-invisible when an assistant fetching it the way an assistant actually fetches — no JavaScript, no waiting — comes away with nothing it can act on.

Not slow. Not ugly. Not badly ranked. Blank.

What is counted

Agent-invisible is not a judgement about a site’s quality. It is a verdict about one fetch, and the code that assigns it is the definition — quoted here rather than described, from tools/fleet_crawl.py:342:

agent_invisible = (None if verdict in ('UNREACHABLE', 'CRAWLER_ERROR')
                          else verdict in ('DEAD_OR_ERROR', 'JS_ONLY', 'EMPTY'))

So a site counts as agent-invisible when one fetch, running no JavaScript, produced an error or a 4xx/5xx (DEAD_OR_ERROR), fewer than 400 characters of text with scripts on the page (JS_ONLY), or fewer than 400 characters and no scripts either (EMPTY).

And two verdicts are counted as neither. When our crawler could not reach a host (UNREACHABLE) or failed on its own malformed input (CRAWLER_ERROR), the value is NULL, not false — we could not attribute it, and calling those sites fine would be as wrong as blaming them. The rate below is therefore invisible over JUDGED, and the sites we could not judge are shown beside it rather than folded away. That distinction is the whole of the correction this page carries at the foot.

What the crawl found

Corrected 2026-09-01. This page said 47.3%, and that number counted sites our own crawler never reached — a timeout or a failed TLS handshake on our side was recorded as a verdict about someone else’s website. Those are now separated out and counted nowhere. A domain that does not resolve is still counted as agent-invisible, because nobody reaches it, human or agent, and that is a fact about the domain. The figure fell by 4.5 points. Six more are counted separately again: our client could not even encode those URLs, which is our defect and not a finding about anyone. The earlier number is left here rather than quietly swapped, because a number that moves without saying so is worth less than one that never moved.

The number that matters is the last one

Nearly half of these businesses are invisible to an assistant. That figure gets the attention, and it should.

But the more useful row is the bottom one. Every site that does expose tools an assistant can call got them from its platform — Wix or Shopify shipping the capability to their tenants. Not one business put it there deliberately.

So this is not a story about businesses being behind. It is a story about the platforms quietly making their own tenants reachable while everyone else stays dark, and nobody choosing to be reachable on purpose.

What it is not

Agent-invisible is not a score, a grade, or a ranking. It is a yes-or-no observation about one fetch, and it is reproducible: point the same request at the same URL and you get the same answer. Where we could not measure something, this documentation says so rather than estimating it.

SNAPSHOT 2026-09-04T01:47:53Z · no figure rendered