EmaIngestBot

1. Identity

Unless the customer has added a note of their own — the next paragraph explains that — every request EmaIngestBot makes carries exactly this User-Agent header:

EmaIngestBot/1.0 (+https://www.puglieseweb.com/crawler)

Match on the product token, EmaIngestBot, rather than on the whole string. A customer can add a short note inside the parenthesised comment from their own console, so that their operations team can recognise the traffic in their logs. It is their field rather than ours: it is capped at 80 characters, brackets, semicolons and control characters are stripped out of it, and it is refused outright if it reads like the name of another crawler — but nobody here reviews it before it goes on the wire, and we would rather you knew that than assumed an approval step that does not exist. What a note can never do is replace the product token, the version or the link to this page.

For example, with a customer note appended:

EmaIngestBot/1.0 (+https://www.puglieseweb.com/crawler; site=example.com)

We do not run the crawler under any other name, and we do not send a browser-like User-Agent to disguise it.

2. What it does, and why

Ema answers a business's inbound email. To answer accurately it needs the business's own published material — opening hours, delivery terms, product details, returns policy — so EmaIngestBot fetches the pages of that business's website and passes the text to Ema.

It requests pages over HTTPS the way an ordinary visitor would. It does not sign in, submit forms, or try to reach anything behind authentication, and it does not place orders. It reads web pages and XML sitemaps and nothing else: a PDF, an image, a video or a JSON endpoint is refused on its content type before the body is downloaded.

A crawl runs only after the customer has proved that they control the domain, and so does the earlier step that lists a site's pages for them to choose from. Two kinds of request do reach a domain before anybody has proved anything, and we would rather name them than have you find them in your log. The first is a single HEAD request, sent when a customer saves a web address on their account or changes the one already there: it asks only whether the address resolves and answers, it gives up after five seconds, and any reply at all — a 404 or a 500 included — counts as an answer. The second is the verification check itself, which is how the proof gets made: a DNS lookup, a read of the token file at /.well-known/ema-site-verification.txt, or a read of your homepage looking for the line we issued. Neither is a crawl. The HEAD request downloads no page at all; the verification reads stop at the first kilobyte of a token file or the first 512 KB of a homepage, and nothing they see is stored, indexed, or used to answer anybody's question. Section 4 says what these two do about robots.txt: they do not consult it.

3. Authorisation: we verify domain control first

The United Kingdom has no commercial text and data mining exception. Section 29A of the Copyright, Designs and Patents Act 1988 covers non-commercial research only. The lawful basis for copying a website at commercial scale is therefore the licence granted by the copyright owner, which for us means the customer. That licence is worth nothing if the customer does not actually control the domain they typed into our onboarding form, so we check rather than take their word for it.

A crawl of a domain begins only once one of the following has been recorded against the account. The check is enforced at the single point that the console's crawl button and the nightly schedule both pass through, so there is no route that starts a crawl around it, and the page-listing step is gated on the same record. Throughout, example.com and www.example.com are treated as one domain:

A DNS TXT record

Published at the apex of the domain, or at _ema-verify.<domain>, containing the token we issue. We look on both names, through public DNS-over-HTTPS resolvers:

; at the apex of the domain
example.com.             IN  TXT  "ema-site-verification=<token>"

; or on a dedicated host
_ema-verify.example.com. IN  TXT  "ema-site-verification=<token>"

A token file

The token we issue, served as plain text from the domain itself. We read the first kilobyte of the file, over HTTPS only, following at most two redirects:

https://example.com/.well-known/ema-site-verification.txt

A token in your homepage

A line we issue, pasted into the homepage of the domain — either as a meta tag in the <head>, or as plain text anywhere on the page. This is the route for a business that has no mailbox at its own domain and no access to its DNS, which is the ordinary situation for a small business whose site is on Wix, Squarespace or GoDaddy. It is also the proof that fits what is being licensed most closely: the other two establish control of the domain, whereas this one establishes control of the content the crawl copies. We read the first 512 KB of the homepage looking for it, which is why the meta tag is the shape we recommend — the <head> is always inside that. The line can be removed once we have confirmed it.

A recorded operator attestation

The fourth route, and the only one a customer cannot complete themselves. Where the authorisation exists on paper rather than in a DNS record — a signed statement of work covering the customer's own estate, say — a member of PuglieseWeb staff records it against the account, and the record names who attested it and where the authorisation is held. It is also how every domain that was already configured for a crawl was cleared when this check was introduced: those customers had authorised the crawl when they engaged us, before there was any token to publish, and the alternative was to stop crawling for all of them overnight without telling them. An attestation is our own say-so and is weaker than a proof the customer publishes, so we still ask for one of the three above.

Verification is recorded against the account, and a crawl reads that record rather than re-checking the proof. We do not re-resolve the DNS record or re-fetch the file before each crawl, and a later check that fails does not remove a verification already recorded — so taking your record down does not, on its own, stop us. There is also a fourth route, which a customer cannot use: a member of our staff can record an attestation against an account whose authorisation is held on paper, naming who attested it and where it is kept, and that is how every domain already configured for a crawl was cleared when this check was introduced. If you want an authorisation withdrawn, tell us — the address is in section 7 — and we will suspend crawling of that domain within one business day, and delete what we hold from it if you ask.

4. robots.txt

We honour robots.txt for our own product token, as a matter of policy, on every domain we read — including a customer's own. If a customer authorises a crawl of their site and their robots.txt disallows us, the robots.txt wins and we read nothing. What we do when we cannot read your robots.txt is deliberately not symmetrical, and it is the part worth knowing: a 404 or a 410 is your site answering that it has no policy, so everything is allowed, which is what RFC 9309 section 2.3.1.3 says it means; anything else — a 5xx, a 403, a 429, a timeout, a refused connection — is your site failing to answer, so we treat the whole domain as disallowed and fetch nothing, per section 2.3.1.4. The sites least able to serve a robots.txt should not be the ones we crawl hardest.

Be clear about what robots.txt is, though. It is not a grant of access. An Allow rule is not a licence to copy a site, and the absence of a Disallow rule is not consent. Our authority to crawl comes from verified domain control and a contract with the copyright owner, never from a file that anyone can serve. We say so here so that nobody is misled about the basis on which we operate.

To stop EmaIngestBot crawling your site, add this to your robots.txt:

User-agent: EmaIngestBot
Disallow: /

To allow it explicitly — useful if your robots.txt disallows unknown agents by default:

User-agent: EmaIngestBot
Allow: /

We fetch robots.txt once, before each crawl begins, so a change takes effect on the next crawl rather than part-way through the current one. Two kinds of request do not consult it, and both are named here rather than left for you to discover: the single HEAD request that checks an address a customer has just saved or just changed, and the fetches that look for a verification token at /.well-known/ema-site-verification.txt or in a homepage. Neither is part of a crawl. The HEAD request downloads no page; the token fetches read only far enough to find the line we issued — the first kilobyte of the token file, or the first 512 KB of a homepage — and keep none of it. Section 2 describes both from the other end.

5. Rate limits and crawl behaviour

Concrete limits, because "we crawl politely" is not something you can put in a firewall rule:

• Requests to a single host are made one at a time within a crawl, and they are paced: at least one second apart during a crawl, and at least half a second apart during the shorter page-listing step that precedes it. The pacing is one setting for the whole service, and it governs each job rather than interlocking separate ones, so a crawl and a page listing started close together are not held apart from each other.
• Every request has a timeout, and it differs by phase: 30 seconds for a page during a crawl, 6 seconds during page listing, 8 seconds for the homepage read that checks a verification token, and 5 seconds for the one-off check that a newly saved address answers. A request that exceeds its timeout is abandoned and the page recorded as failed.
• A crawl stores at most 60 pages, and it makes more requests than that — so if you are reconciling this against your access log, count requests rather than stored pages. A whole-site scan fetches up to about 120 pages before its budget stops it; a crawl of a hand-listed scope fetches up to 60 listed pages, plus up to 20 more read only for the question-and-answer pairs on them. On top of either: one robots.txt, and up to five sitemap requests. The 60-page ceiling is fixed in the product and is not raised by arrangement.
• On HTTP 429 or 503 — the two answers that mean later rather than no — we wait and try the same page at most twice more, doubling the wait each time from the base delay and never waiting longer than 12 seconds. After that the page is recorded as failed and the crawl moves on to the next one. A 404 is an answer rather than a hiccup, and is never retried.
• A crawl stays on the verified domain: links to other hosts are not followed, URLs carrying a query string are not fetched at all, and a response body is abandoned once it passes 2 MB. At most five crawls of a domain can be started from a customer's console in a day, and a scheduled run counts towards that five. Our own staff are not subject to that cap — a support call is not an abuse case — so if you want a ceiling you can rely on, it is the 60 pages, not the five runs.

If that is still more than your infrastructure wants to absorb, tell us. We can cut the scope for your domain — a shorter list of pages, or crawling only when someone asks for it rather than on a schedule — and we can slow the crawler down, although the pacing is one setting for the whole service rather than a per-domain dial. Either is faster than blocking us, and easier for both of us than an incident.

6. Verifying that traffic really is us

The User-Agent header proves nothing on its own. Any client can send any string, and crawlers that would rather not be identified routinely borrow the identity of ones that would. If you are seeing aggressive traffic claiming to be EmaIngestBot, it may well not be ours — and blocking on the string alone stops the polite bots and none of the impolite ones.

If you need to know for certain, send us the affected domain, the timestamps with their time zone, and the source IP addresses from your logs. Every crawl we run writes a row of its own: when it was requested and by whom, the host it was for, whether that host was verified at the time, and which of the four proofs was on file. So we can tell you whether we were crawling your domain in that window and on what authority. What we cannot do is match your log lines to ours one request at a time — we do not record the address each request goes out from — so the answer you get covers the window rather than the individual line.

We do not currently publish a list of egress IP addresses for allowlisting. If your environment needs one, ask us.

7. How to stop us entirely

Three routes. Any one of them is enough:

• robots.txt — add the Disallow block from section 4. It takes effect on the next crawl.
• A firewall, CDN or WAF rule — block the product token or the source addresses. We will not try to work around a block: we do not rotate addresses, change our User-Agent, or retry from elsewhere in order to get past one.
• Email us — we will suspend the crawl and, if you ask us to, delete the content already ingested from your domain.

8. Related pages

Two other pages answer the questions that usually follow this one:

• Trust and Security — Hosting, sub-processors, encryption, retention, breach notification and vulnerability reporting.
• Privacy Policy — What personal data we collect, the lawful bases we rely on, and your rights under UK GDPR.