The research index.

The research index measures one thing across a cohort of company websites: how much of what a site says about itself an answer engine can actually read and quote. Nothing has been published yet, and when something is, it reports the cohort in aggregate.

If you saw VikramaResearch in your logs

That was this crawler taking one reading of your homepage. Every number below is read from the crawler source, not described from memory.

The user agent

VikramaResearch/1.0 (+https://vikrama.studio/research; measuring answer-engine readiness; obeys robots.txt)

What it requests

Eight requests at most, in this order, and then it leaves. Every one of them after the first is asked for only if your robots.txt allows it:

1. /robots.txt
2. The sitemap named on the first Sitemap: line of your robots.txt, or /sitemap.xml if no line names one. One request. A sitemap you named is fetched whatever your rules say, because that line exists to point crawlers at it; the guessed /sitemap.xml is only fetched where your rules allow it. If it is a sitemap index, the children are counted as child sitemaps and never as pages, and none of them is fetched.
3. /llms.txt
4. One path that should not exist, /vikrama-teardown-probe-<timestamp>, to see whether your server answers a missing page with a real 404 or with a 200. The path carries the word teardown because the probe code is shared with the teardown crawler. A research run really does request that path.
5. The www or apex counterpart of your homepage, redirects not followed.
6. Your homepage, redirects not followed.
7. Your homepage again, following redirects.
8. Your homepage once more, read for the measures below.

No other page of your site is requested. There is no crawl. The default pause between requests is 1500 ms. A site that disallows everything sees two requests, not eight: your robots.txt, and the sitemap you declared in it.

What is measured

From the homepage HTML as served, read with regular expressions. No browser, no DOM, no JavaScript executed:

• JSON-LD: how many blocks, how many failed to parse, and which types appear.
• Meta description: whether the tag is there, whether it says anything, how many characters.
• How many h1 opening tags, the matched-pair count, and the order of heading levels.
• The page title, the host that answered, the redirect chain, status, bytes and response time.
• A platform guess from 16 published signals, for example a Shopify header or a Webflow asset host. The signal that matched is recorded with the guess.
• Whether /robots.txt, /llms.txt and a sitemap exist, and what your server does with a missing page.

Anything your robots.txt forbade is recorded as not measured rather than as a zero or a no. If your rules disallow /llms.txt, the record does not say you lack one; if they disallow /, there are no homepage measures at all, and the record says why.

Your homepage HTML is not written to disk. Neither is any other page body. The record for a domain is these counts, the status lines, and the probe results.

How it treats robots.txt

The user agent string claims the crawler obeys robots.txt, so it does. Rules are matched against the token before the slash, VikramaResearch, and against User-agent: *. Per RFC 9309, a 4xx on robots.txt means no rules apply. A 5xx, or a 429, means the whole site is treated as disallowed and the homepage is not read. If your rules disallow /, the record says so and the homepage is never fetched. That is a result, not a failure: the question was what a crawler can read, and the answer was nothing.

A robots.txt that errors and one that says no are recorded as two separate things. A site whose robots.txt was simply down is never reported as having chosen to block answer engines.

Timeouts and retries

15 seconds to connect, 30 seconds for the whole request. Six redirect hops at most. One retry, and only where a retry can help: a timeout or a 5xx. A 4xx is an answer rather than a failure, so it is never retried.

Removal

Email hello@vikrama.studio with your domain and ask for removal. The record for that domain is deleted within 10 business days, the domain is dropped from the cohort, and it is not measured again. No reason needed. If a report has already gone out, the next edition drops the domain.

The other crawler is the teardown, which does read more than one page. It is documented on the teardown page.