SpotsBot

Spots keeps a directory of HMC-certified outlets in sync. To do that we fetch pages from halalhmc.org once a night.

What it does

• Reads the public sitemap and the public by-name outlet listing.
• Fetches outlet pages on a rolling cycle — roughly a seventh of the catalogue each night, about 100 pages, at one request per second on a single connection.
• Sends conditional requests, so unchanged pages cost almost nothing.
• Re-reads robots.txt at the start of every run and obeys it.
• Stores only factual listing data: name, address, phone, category, certification status.

What it doesn’t do

• It doesn’t submit forms, follow links off-site, or touch anything behind a login.
• It doesn’t republish HMC’s pages — every spot links back to its certificate.
• It never runs faster than one request per second, whatever the time of day.

User-Agent

SpotsBot/1.0 (+/about-bot; hassaanc13@gmail.com)

Contact

If you’d like the crawl slowed, changed or stopped, email hassaanc13@gmail.com and we’ll action it the same day. A Crawl-delay or Disallow in your robots.txt also works — we read it every run.