robots.txt, always
Rules for SyntoraDataBot first, otherwise those for *. If robots.txt errors or doesn't answer, we don't crawl until it does.
SyntoraDataBot/0.2
Our crawler
SyntoraDataBot reads the public websites of companies listed in official business registers, so that our company records are accurate and current. It follows robots.txt, goes slowly and says who it is.
Identity
- User-Agent
SyntoraDataBot/0.2 (+https://syntoradata.com/bot)- robots.txt token
SyntoraDataBot- IP address
162.55.129.55- Reverse DNS
crawler.syntoradata.com
How it behaves
5 seconds, or your Crawl-delay
At least 5 s between two requests to one site. A Crawl-delay is honoured in full, however long.
25 pages, two links deep
The homepage and its about, contact, team, careers, legal and privacy pages. Never more than 25 per site.
HTML only, 2 MB cap
No PDFs, images or downloads. No forms, logins, paywalls or CAPTCHAs, ever.
Backs off on trouble
HTTP 429, 5xx and timeouts slow it down; five in a row pause the site for 7 days.
One IP, no proxies
Every request comes from one fixed, identified address with a matching reverse DNS.
What it visits
- Only websites of companies we know from official registers, global identifier sets (GLEIF, Wikidata) or licensed company datasets, worldwide, and only the company's own domain.
- Never social networks, search engines, directories or other third-party sites, and no government or university sites.
- Team, about and contact pages are revisited every month, so people who have left drop out quickly; homepages about every 3 months, other pages every 6 months.
Finding a company's website
Some registers and licensed datasets give the website; most registers do not. To link a registered company to its site we may then try a few likely domains based on its name. On each one we read robots.txt first, then at most the homepage and one legal or contact page.
A site is linked to a company only when it shows the company's number, or its name with the registered postcode. A site that turns out not to be that company's is not visited again.
Block it or slow it down
Block it completely
User-agent: SyntoraDataBot Disallow: /
Keep it out of some pages, or slow it down
User-agent: SyntoraDataBot Disallow: /internal/ Crawl-delay: 30
Changes to robots.txt are picked up within 24 hours. For anything faster, or if you think the crawler misbehaved, write to privacy@syntoradata.com with your domain and we will stop crawling it straight away.
Verify it is really us
A request is from SyntoraDataBot only if it comes from 162.55.129.55, whose reverse DNS is crawler.syntoradata.com, which in turn resolves back to the same address:
host 162.55.129.55 # -> crawler.syntoradata.com host crawler.syntoradata.com # -> 162.55.129.55
What we keep
Company facts the site publishes: switchboard numbers, general mailboxes, addresses, VAT numbers and company social profiles. Where the site publishes them, the work contact details of named staff, under strict rules and not shown to anyone.
Every fact keeps the page it came from. See data points and our privacy notice; anyone can see or remove what we hold about them.