IntentRouteBot

Last updated: 11 October 2026. Change log at the bottom of this page.

Who we are

IntentRouteBot is the crawler of IntentRoute (intentroute.net), operated by Oleksandr Kharybin, United Kingdom. A dedicated company will take over in 2027. IntentRoute is a neutral public index of the files that websites publish for AI agents. We check whether these files exist, whether they are valid, and how they change over time. We do not sell placement, and we do not route or resell traffic.

• User-Agent: IntentRouteBot/0.1 (+https://intentroute.net/crawler). The version number may change; the robots.txt token IntentRouteBot never does.
• Verify it is us: we crawl only from the IP addresses listed in /crawler/ips.json. Each address has reverse DNS under intentroute.net with a matching forward record. Check: host <ip> returns a name ending in .intentroute.net, and host <that name> returns the same IP. Anything else using our name is not us.

Why your site was visited

Your host is on a public list of popular websites (Majestic Million, CC BY 3.0), in a public registry of agent services (for example the official MCP Registry), or it publishes one of the files below. We visit about once a week.

What we fetch — and nothing else

One GET request per file location, about 12–14 requests per site per week. Most of them return 404, which is expected and fine.

The crawler does not follow links, does not run JavaScript, does not store or send cookies, never submits forms, never logs in, never tries to bypass a block or a CAPTCHA, and never writes anything to your site. robots.txt is not access control, and we treat a block as final.

How it behaves

• robots.txt (RFC 9309): we follow the IntentRouteBot group, or User-agent: * if there is none, per host and per subdomain. A 4xx means no restrictions. A 5xx, 429, timeout, more than 5 redirects or a file over 500 KiB means we do not crawl that host this week.
• Crawl-delay is honoured at any value. Otherwise one request at a time per site, at least 2 seconds apart.
• 429 or 503: we stop for that site, honour Retry-After, and come back next week at the earliest.
• Conditional requests: we send If-None-Match / If-Modified-Since, so an unchanged file costs you a 304. We accept gzip and brotli.
• Size: we stop reading a file after 10 MB and discard it.
• Time window: each run starts on Monday around 06:00 UTC and usually finishes within 24 hours.

What we do with the files

• We keep an exact copy of each fetched file with its fetch date and a SHA-256 digest, so that a later change can be shown precisely.
• We publish, per site: which files exist, whether they are valid, and when they changed, plus aggregate statistics. Published results are under CC BY 4.0, cite as host, observed date, SHA-256, IntentRoute (intentroute.net), CC BY 4.0.
• We do not republish raw copies of your files as a dataset. We do not use them to train AI models. Validity checks are plain code; no fetched profile file is sent to an outside AI service.
• Raw copies are kept for as long as they back a published change history; you can ask for deletion (below).

Personal data (UK GDPR)

We do not look for personal data. Some files contain contact names or e-mail addresses (for example the contact field of an OpenAPI description or an agent card). The controller is Oleksandr Kharybin (United Kingdom), contact@intentroute.net. Lawful basis: legitimate interests — publishing an independent, dated record of public machine-readable files. You can ask for access, erasure or objection at contact@intentroute.net; we answer within one month. You may complain to the Information Commissioner's Office (ico.org.uk).

How to opt out

• robots.txt — the standard way. Block everything:

  User-agent: IntentRouteBot
  Disallow: /

  or only some files, for example Disallow: /.well-known/. It takes effect at our next visit, within 7 days at most.
• E-mail — faster, and also removes history. Write to contact@intentroute.net with the subject opt-out <your-domain>, from an address at that domain (or ask us for a DNS TXT check). We acknowledge within 1 business day and exclude the domain within 2. On request we also delete its published results, stored copies and history within one month; we keep only the domain name on a suppression list so that we do not crawl it again.

Contact

Problems with the crawler: contact@intentroute.net. Please include the time (UTC), the source IP and the User-Agent from your logs; we reply within 2 business days.

Change log

• 11 October 2026 — first publication.