UnburyBot

The crawler that builds the UNBURY index. UNBURY is made by Galactrex.

How it crawls

• robots.txt first. Its token is UnburyBot. Without an UnburyBot group it follows User-agent: *.
• One request at a time per site, a few seconds apart, or your Crawl-delay if that is longer.
• A short, hand-picked list of sites, with a page limit for each one.
• Page rules respected: noindex, nofollow and canonical links.
• User agent: UnburyBot/0.1 (+https://unbury.net/bot)

Block it

User-agent: UnburyBot
Disallow: /

Pages it already indexed are removed on its next visit.

Contact

Questions or problems: [email protected]