Why our crawler is deliberately slow
ColdDeck reads a few pages per site at roughly human speed. That decision cost throughput and bought something worth more.
ColdDeck needs to know things about a business before we contact them: whether they can take a booking online, whether there is any way to reach them that is not a phone call, whether the contact form actually posts anywhere.
The fast way to find that out is to hit every page of every site as quickly as the network allows. We do the opposite, on purpose.
What it actually does
A few pages per site — home, contact, about, and whatever looks like a booking or services page. Not the whole site. Sequential, with a delay, at roughly the pace a person clicking around would generate.
If a site owner looked at their access log, the visit should look like a curious human rather than an incident.
Three reasons, in order of how much they mattered
It is somebody's production server. A small business site is often on shared hosting with a database that falls over under load. Hammering it to decide whether to send a sales email is a genuinely rude thing to do, and the fact that it is technically permitted does not make it fine.
The signal does not need the volume. This is the practical argument. Nothing we detect requires deep crawling. A booking widget is on the home page or the services page. A contact form is on the contact page. Crawling forty pages to find something that lives on four is throughput spent for nothing.
It survives. Aggressive crawling gets you blocked — by the site, by Cloudflare, by the whole IP range. A polite crawler runs indefinitely. Over a month, the slow one collects more data than the fast one, because the fast one spends half the month rate-limited.
That last point is the one people miss. Politeness is not just ethics here; it is the higher-throughput strategy on any horizon longer than a day.
The part that is actually load-bearing
Every detection stores the URL it fired on.
That sounds like a logging detail and it is the reason the whole system works. When the score says 34 and the email says "you have no way to take a booking online", both statements point at a specific page that was read at a specific time. If the business owner replies "we do have online booking", we can go and look, and usually find it is on a page we did not read or behind a JavaScript widget that did not render.
Then the rule pack gets fixed. That loop is only possible because the evidence was kept.
A crawler that discards what it saw can never be debugged, only re-run and hoped over.
What it cost
Throughput, straightforwardly. A niche takes hours rather than minutes.
That turned out not to matter, because the human approval step downstream is the real bottleneck. Nobody is reading and approving ten thousand emails a day, so producing candidates faster than a person can review them optimises a stage that was never constrained.
Most pipelines have a step like that — one where speed feels like progress and buys nothing, because something slower sits behind it.
ColdDeck is written up in full as a rule pack, not a model. The same evidence-first reasoning shows up in our extraction work.