For publishers
NupdateBot
Nupdate is a free, non-commercial hobby project that links public nuclear-sector news and announcements. This page explains what its crawler does.
How to recognise it
Requests carry this User-Agent:
NupdateBot/1.0 (+https://nupdate.nl/bot)
What it reads
- Public RSS and Atom feeds, and the public page of a site where it looks for a feed link.
- It does not log in, does not get past paywalls, bot challenges or access controls, and does not use search-engine news feeds.
How politely it behaves
- It reads your
robots.txtfirst and follows it. If robots.txt cannot be read, it does not crawl. - A given source is requested at most once every 30 minutes; most sources are polled once an hour or less often.
- It sends
ETagandIf-Modified-Sinceso unchanged feeds cost almost nothing. - After an error it backs off exponentially, and only a few sources are fetched at the same time.
- A source that stops working is marked as such and dropped from the active list; it is not retried in a loop.
What it keeps
Only the title, a short excerpt, the date, the language and the link to your original. Full articles are never stored or shown, and every item on Nupdate links to its publisher.
Opting out
Add this to your robots.txt and the crawler stops at its next visit:
User-agent: NupdateBot Disallow: /
This is the supported way to opt out. No email address is published on this site.