For whoever runs the site

The AnteSEO crawler in your logs

If you have found x-anteseo in your access logs, this page is the whole answer: what we are, why we came, how to recognise us, and how to slow us down or turn us away.

Why we were on your site

Somebody bought a report about that domain. AnteSEO sells a one-off written report that says which pages a website is missing, and producing one means reading the site the way a search engine reads it. We do not crawl sites nobody has asked us to crawl, we keep no index, and we will not come back unless somebody buys another report on the same domain.

Our terms require the buyer to own the site or be authorised to act for whoever does. If you believe nobody with that authority asked for this, tell us the domain and we will say when the crawl was bought and refund it: hello@anteseo.com.

How to recognise us in a log

Match the header, not the user-agent. Every request we make carries x-anteseo with this page's address as its value, and that is the field to write an allowlist or a block against.

Header, on every request: x-anteseo: https://anteseo.com/bot
User-agent, plain requests: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/152.0.0.0 Safari/537.36
User-agent, rendered pages: The installed Chromium's own string, with the word Headless removed. It moves with the browser build, so there is no fixed string to match.
Methods: GET only. We never submit a form, never POST, and never sign in to anything.
Requests, at most: Two robots.txt, because it is read once to plan the crawl and once by the reader that opens the pages. Up to 5 guesses at where your sitemap lives, if the file names none, and up to 40 sitemap files after that. Four requests to check which of your four front doors answers, one or two more when the domain was first typed, and up to 100 pages for a report, or 5,000 when the site's owner follows it on a package.

We do not announce ourselves as a bot in the user-agent, and that is deliberate

The product measures what a browser and a search engine can see, so we read the site in a real Chromium. For as long as the user-agent said AnteSEOBot, bot filters turned us away from crawls the site's own owner had paid for and authorised. The name moved into a header nothing blocks by accident, which is why the header above is the thing to match: you cannot allowlist something that has no name.

How we behave

We read robots.txt and obey it. An address it disallows is never requested at all: not fetched and discarded, never asked for. Three requests come before it or beside it, and the note below names them.
We obey the User-agent: * group, and only that one. A rule addressed to us by name has no effect, which is covered below.
We pause 250ms between pages by default, and longer if your Crawl-delay asks for it, up to 5 seconds.
We open at most 100 pages of a site for a report, and at most 5,000 for a site its owner follows on a package, best-first, however large the site is. On a package each page is a plain request first, and Chromium only for a page that refuses one.
We never touch anything behind a login, and nothing that requires a cookie or a session.
If requests keep failing we stop early rather than hammering a server that is clearly refusing us.

Three requests do not wait for robots.txt, and we would rather name them here

Your home page is fetched before the file can have been read, and is never ruled out by it afterwards: it is the one address a report cannot do without. The four requests that work out which of the bare name, www., http and https is your real front door do not read it either. Nor does the single check that the site answers at all, which runs the moment somebody types the domain, before any money has moved. Everything after those obeys the file, and the terms say the same thing in the same words.

What we keep is the rendered HTML of the pages we read and the text of the site as one document, because those are the evidence behind the report and what makes a finding checkable afterwards. The privacy notice says how long that is kept and what else it is used for, which is nothing.

Slowing us down

Add a Crawl-delay to the * group in your robots.txt and we will take it, up to 5 seconds. Beyond that we cap it: a site asking for a minute between pages is asking for longer than a report is worth waiting for, and we would rather be honest about the ceiling than quietly ignore the file.

User-agent: *
Crawl-delay: 5

Turning us away

A Disallow in the * group keeps us out, and it will keep search engines out too, which is the thing to weigh before using it.

User-agent: *
Disallow: /

Blocking us by name in robots.txt will not work

We obey the * group and no other, so a User-agent: AnteSEOBot block is read by nothing. That is the stricter behaviour rather than the looser one, since a permissive rule addressed to us by name is hypothetical and a restrictive * is not. It does mean that a rule written at us alone will not turn us away. To refuse us specifically and keep search engines, block the header at your server or edge instead.

In nginx, refusing the header is two lines in the server block:

if ($http_x_anteseo) { return 403; }

If you would rather we simply never came back, send us the domain and we will refuse further sales on it: hello@anteseo.com. That is a better outcome for both of us than a block we would have to discover by being refused.

What a block costs the site owner

Worth saying plainly, because it is the reason to think before blocking rather than a reason not to. Whatever turns us away is usually deciding the same thing about search engines: we arrive as an ordinary browser, from one address, at a few pages a minute. If we cannot read the site, that is itself a finding, and it is one the owner generally wants to know about.

When a crawl is refused outright we cannot produce a report, and we refund the buyer in full.