NafasiBot
NafasiBot reads public scholarship, fellowship and admissions pages so African students can find opportunities and apply on the official site. It identifies itself as NafasiBot/1.0 (+https://nafasiafrica.com/bot; bot@nafasiafrica.com).
Rules it follows
- It obeys robots.txt, including Crawl-delay. If robots.txt cannot be read, it does not fetch.
- At most one request every 5 seconds to any one site, usually far fewer: most pages are checked once a day.
- It only visits sites on a reviewed list, and a 401 or 403 stops it until a person reviews the site.
- It keeps facts such as the title, deadline and the link to apply. Students are sent to your page to apply.
Asking it to stop
Add this to your robots.txt:
User-agent: NafasiBot Disallow: /
Or email bot@nafasiafrica.com with your domain. Takedown requests are handled by a person and remove the source and its listings.