SyntoraData

SyntoraDataBot

SyntoraDataBot is the web crawler of SyntoraData (Syntora Ai). It reads the public websites of companies listed in official business registers, so that our company records are accurate and current.

What it visits

• Only websites of companies we already know from official registers, and only the company's own domain: never social networks, search engines or third-party sites.
• The homepage and the pages it links to about the company: about, contact, team, careers, legal/imprint, privacy. At most 25 pages per site, two links deep.
• HTML pages only, up to 2 MB. No PDFs, images or downloads, no forms, no logins.
• Each page is revisited about every 3 months (homepage) or 6 months (other pages).

How it behaves

• robots.txt is honoured: rules for SyntoraDataBot first, otherwise those for *. Disallowed pages are never fetched.
• At least 5 seconds between requests to the same site, or longer if you set a Crawl-delay.
• It slows down on errors (HTTP 429 / 5xx or timeouts) and stops visiting a site for 7 days after five errors in a row.
• It never hides its identity: one fixed IP address, the User-Agent above, no proxies.

How to limit or block it

Block it completely:

User-agent: SyntoraDataBot
Disallow: /

Keep it out of some pages, or slow it down (seconds between requests):

User-agent: SyntoraDataBot
Disallow: /internal/
Crawl-delay: 30

Changes to robots.txt are picked up within a week. For anything faster, or if you think the crawler misbehaved, write to privacy@syntoradata.com with your domain: we will stop crawling it straight away.

What we keep

Company facts the site publishes (switchboard phone numbers, general mailboxes such as info@, addresses, VAT numbers, links to company social profiles) and, where the site publishes them, the work contact details of named staff. Every fact keeps the page it came from. People can see and remove what we hold about them: remove my data.