Our web crawler
What MyTownRocksDiscovery reads and how to stop it
Last updated: 5 November 2026
What it is
MyTownRocksDiscovery is the web crawler run by My Town Rocks™. It helps the MTR team keep our town websites up to date with what's on locally: events, activities and places to visit.
If you have seen it in your website's logs, it identifies itself with this user agent:
MyTownRocksDiscovery/1.0 (+https://www.mytownrocks.co.uk/bot)What it reads
The crawler only visits websites chosen for a town by the MTR team, the town's franchisee or the site's own organiser, such as a venue, a club, an event organiser or a council what's-on page. On each of those sites it reads your robots.txt file, your terms or privacy page, so we can check how your content may be used, and the event and place pages we have picked.
From those pages it takes facts: names, dates, times, prices, opening hours and addresses, and the link to the page they came from. We never republish your wording or your images. Anything we publish is written in our own words from those facts.
How it behaves
- It reads your robots.txt file before anything else and follows it, including any Crawl-delay you set.
- If your robots.txt has no section for MyTownRocksDiscovery, it follows the rules you set for all crawlers (
User-agent: *). - It spaces out its requests so its visits stay light.
- It only reads public pages. It never signs in, fills in forms or gets past CAPTCHAs.
- It never visits closed or members-only platforms, such as Facebook, Instagram or Nextdoor.
How to stop it
To stop the crawler visiting your website, add these two lines to your robots.txt file:
User-agent: MyTownRocksDiscovery
Disallow: /To keep it out of one part of your site only, put that path after Disallow instead, for example Disallow: /members/.
The crawler keeps a copy of your robots.txt for up to a day, so a change can take up to a day to apply.
You can also email crawler@mytownrocks.co.uk with your website address. We will stop visiting your site and remove the details we took from it.
Questions
If you have a question about the crawler, email crawler@mytownrocks.co.uk and the MTR team will get back to you.