/ BOT
AISearchReadyBot
The crawler that performs every audit on aisearchready.io. Visits your site only when a user has explicitly pasted your URL into our scanner. No speculative discovery, no opportunistic sitemap walking.
/ QUICK FACTS
- User-Agent
AISearchReadyBot/1.0 (+https://aisearchready.io/bot; SEO audit)- Rendered variant
AISearchReadyBot/1.0 (+https://aisearchready.io/bot; SEO audit; rendered)- What triggers a visit
- A user pastes your URL into the scanner. Watcher subscribers also trigger a weekly re-scan of their own site.
- Concurrent connections
- One per host. Never parallel.
- Honours robots.txt
- Yes — including
Crawl-delayand per-group rules. - Honours noindex / X-Robots-Tag
- Yes — surfaced as a finding in the report, never indexed.
- Crawls beyond the given URL
- Single-page scans: no.
Deep Audit: sitemap-discovered URLs, max 10 pages, 3 concurrent. - JavaScript rendering
- Optional. Static-HTML fast path by default; escalates to headless Chromium when the page looks like a JS app shell.
What we read
- • The requested URL's HTML
- • Sidecar files at well-known paths:
/robots.txt,/llms.txt,/llms-full.txt,/.well-known/agent-auth - • Sitemap files (Deep Audit only):
/sitemap.xml,/sitemap_index.xml - • HTTP response headers (security, caching, compression)
- • TLS certificate metadata (issuer, validity, protocol version)
What we do NOT do
- ✗ Discovery crawling. We never visit a URL without explicit user intent.
- ✗ Form submission. We send only GET / HEAD requests.
- ✗ Authentication bypass. We don't attempt to access login-walled content.
- ✗ Training data collection. Captured HTML is used to produce one audit report; not reused for ML training.
- ✗ Aggressive retry. One attempt per URL with a 12-second timeout. No backoff hammering.
How to block AISearchReadyBot
We honour robots.txt fully. Drop this into your /robots.txt to stop us visiting (effective on the next scan attempt):
User-agent: AISearchReadyBot Disallow: /
We'd prefer you didn't — visits only happen when someone is trying to audit your site, and blocking us means their report can't be produced. But the directive is authoritative; we won't override it.
Contact
Found us misbehaving — high request volume, ignored robots.txt, anything else? Email below. We'll respond within one working day and can provide a log of every request our bot has made against your site.
bot@aisearchready.io/ GOING DEEPER
The standard we hold ourselves to
We publish a six-rule code of conduct for AI crawlers — the standard we follow and the standard we'd like other AI operators to follow. If your bot meets all six, tell us and we'll list it.
Read the AI Crawler Code of Conduct