RolestashBot
RolestashBot is the crawler that reads public job postings for Rolestash, so people can find jobs they fit. It reads job pages the way search engines do, and it's easy to turn away.
How it behaves
-
It identifies itself as
RolestashBot/1.0 (+https://rolestash.com/bot). - It reads your robots.txt first and obeys it, including
Crawl-delay. - It asks for one page at a time from each site, at least two seconds apart.
- It finds job pages through your sitemaps and reads the job information you publish for search engines.
- If your site answers 429 or 503 it waits, and then stops for the day. If your site blocks it, it stops.
- It removes recruiters' names, email addresses and phone numbers before anything is stored.
Turning it away
Add this to your robots.txt, and RolestashBot won't read your site:
User-agent: RolestashBot
Disallow: /
To keep it to your job pages only, disallow the rest in the same way.
Contact
Questions, or a posting to remove: email support@rolestash.com.