/ CODE OF CONDUCT

What AI crawlers should look like.

Six rules. The standard we hold AISearchReadyBot to, the standard we'd like every AI-tool operator to follow, and a public yardstick for site operators to measure any bot against.

AI crawlers are multiplying. Most of them ship without naming themselves, without a working bot-page, without honouring Crawl-delay, and without an opt-out path. That's not OK, and the industry is starting to push back. We're publishing what we think the floor should look like.

1. 01

   Identifiable

   / THE COMMITMENT

   Ship a unique User-Agent that points at a real, plain-language documentation page. The URL in the UA string returns HTTP 200 and explains who you are.

   / WHY IT MATTERS

   Most AI crawlers in 2026 still ship under generic Mozilla/5.0 strings or use UA strings whose linked URL 404s. That's the easiest test to fail, and the loudest red flag to a site operator inspecting their logs.

   WE MEET THIS

   Our UA is AISearchReadyBot/1.0 (+https://aisearchready.io/bot; SEO audit) and /bot is a real page with all the facts a site operator needs.
2. 02

   Authenticated

   / THE COMMITMENT

   Ship a verifiable cryptographic identity following the Agent Auth Protocol. Sites should be able to actively verify your bot is who it claims to be — not trust the User-Agent string at face value.

   / WHY IT MATTERS

   The User-Agent header is a hint, not a proof. Anyone can spoof "AISearchReadyBot/1.0" from a script. Sites need a way to verify cryptographically. That layer is now possible — and crawlers that don't adopt it are choosing to be less trustworthy than they could be.

   COMMITTED

   We're working with agentauthprotocol.com and our sibling product AgentPassport. Reference implementation goes live alongside AgentPassport's v1 launch. Sites that hit our verification endpoint will get a signed JSON response confirming the bot is ours.
3. 03

   Bounded

   / THE COMMITMENT

   Concurrency capped per host. Strict honouring of robots.txt against the AISearchReadyBot UA on any URL the user did not explicitly paste. Hard timeout on every request. No retry-without-backoff.

   / WHY IT MATTERS

   The 90% of AI crawler complaints from site operators are about volume, not principle. A crawler that hammers a site at 50 req/sec is a denial-of-service event the operator did not consent to. Strict bounds are the cheapest fix.

   WE MEET THIS

   Single page scans hit at most one URL on the host, plus 4 well-known sidecar files (robots / llms / llms-full / agent-auth). Deep Audit caps concurrency at 3 across distinct sampled pages, all with a 12-second timeout. Sitemap-discovered URLs in a Deep Audit are gated through our robots.txt parser against the AISearchReadyBot UA before sampling — disallowed URLs are skipped and logged. URLs the user explicitly pastes are treated as consent and audited regardless (the user is most often the site owner asking us to check their own page).
4. 04

   Auditable

   / THE COMMITMENT

   Every request we make is logged with a timestamp, the requesting scan-ID, and the reason. Site operators can request a log of all activity our bot has made against their host within 24 hours.

   / WHY IT MATTERS

   When a site operator sees suspicious traffic in their logs, "email us, we'll get back to you" is fine for one-off incidents but unsustainable at scale. A self-serve audit-log endpoint scoped to the operator's own domain is the right pattern, and we'll ship one.

   PARTIAL — IN PROGRESS

   The logging exists internally. The operator-facing export endpoint is on the roadmap before public launch — file path: emailing bot@aisearchready.io works today and we hand-export within 24h. We commit to making it self-serve before opening the spec to other operators.
5. 05

   Revocable

   / THE COMMITMENT

   A single robots.txt directive turns the crawler off site-wide, effective immediately on the next attempted visit. Confirmation of opt-out provided in writing within 24 hours of request.

   / WHY IT MATTERS

   If a site operator wants a crawler gone, the path from intent to enforcement should be one line of text and zero round-trips through customer support.

   WE MEET THIS

   The directive is documented at /bot:

   User-agent: AISearchReadyBot
   Disallow: /

6. 06

   Useful

   / THE COMMITMENT

   Never crawl a site without explicit user intent. No speculative discovery, no opportunistic sitemap walking, no "we found you in a public dataset and decided to scan." Every visit traces to a user action.

   / WHY IT MATTERS

   A meaningful chunk of the negative perception around AI crawlers comes from operators discovering they were crawled at scale without ever opting in, and finding the captured content monetised downstream. The simplest opt-out is "we don't do that."

   WE MEET THIS

   AISearchReadyBot visits only when (a) a user pastes a URL into our scanner or (b) a Watcher subscriber's own site is due for its weekly re-scan. We don't maintain a discovery crawler. We don't sell the captured HTML for training data. We don't share it across customer scans.