Our crawler

Who we are

Item.ie is an independent Irish price comparison site. We list products from Irish retailers side by side, show the current price at each shop and send shoppers to the retailer's own product page to buy. We sell nothing and take no payment from shoppers.

How the crawler identifies itself

Where the crawler names itself it sends the User-Agent "item.ie-bot/1.0 (+https://item.ie/bots)". The robots.txt product token is "item.ie-bot". Some retailers are read through their own published apps or feeds, and those requests carry the identity that feed expects rather than ours.

robots.txt

On every shop where the crawler identifies itself, Disallow rules in robots.txt are honoured before any page is requested: a forbidden URL is never fetched, and each one we decline is recorded on our side. A declared Crawl-delay is honoured unless a shop has agreed a different pace with us.

Pacing

Requests to a shop are spaced with a minimum delay and a small concurrency cap, set per shop. We read listing pages and product pages only; carts, accounts, checkout and site search are never requested.

When a shop says no

A block, a challenge we cannot pass, or a written request to stop is treated as an instruction. The crawler stops that shop immediately, does not retry or change its identity, and the shop stays paused until a person has spoken to the retailer.

How to opt out

Add a robots.txt group for "item.ie-bot" with the paths you do not want read, or write to us and we will pause the shop while we talk. Either route works; we act on both.

Contact

Write to [email protected]. A request to stop is acted on the day it arrives.

More about the site: about Item.ie and data accuracy.