A rule pack, not a model

ColdDeck finds businesses in a niche, reads a few pages of each site, and decides which ones are worth contacting. The obvious way to build the deciding part is to hand the page to a model and ask it to score the prospect out of ten.

It does not do that. The model reads the page and reports what it saw. A deterministic rule pack turns those observations into a score.

That line — between reading and judging — is the most consequential design decision in the system.

The model is good at "is there a booking widget on this page". It is unreliable at "how much does that matter", and only the second one is a decision you have to defend.

Two different jobs

Reading is perception. Given a page, is there a contact form? Does it post anywhere? Is there a phone number and nothing else? Is the site served over HTTPS? These are questions about what is present, and a model handles messy real web pages far better than a pile of selectors ever did.

Judging is policy. A business with no online booking is worth more to us than one with no chat widget. A broken contact form matters more than a missing mobile viewport. Those weightings are a claim about our own business, and they change as we learn what converts.

Collapsing both into one prompt produces a number that cannot be explained, cannot be replayed, and moves when the provider updates the model.

What the rule pack looks like

type Signal =
  | { kind: "no_online_booking" }
  | { kind: "no_contact_form" }
  | { kind: "contact_form_posts_nowhere" }
  | { kind: "no_mobile_viewport" }
  | { kind: "no_ssl" }
  | { kind: "phone_only_contact" }

type Rule = {
  signal: Signal["kind"]
  points: number
  because: string        // shown to a human, verbatim
}

const RULES: Rule[] = [
  { signal: "contact_form_posts_nowhere", points: 30,
    because: "Their contact form silently drops enquiries" },
  { signal: "no_online_booking", points: 25,
    because: "No way to book without phoning" },
  { signal: "no_contact_form", points: 15,
    because: "No way to reach them except a phone number" },
  { signal: "phone_only_contact", points: 10,
    because: "Phone is the only contact route" },
  { signal: "no_mobile_viewport", points: 8,
    because: "The site is not usable on a phone" },
  { signal: "no_ssl", points: 5,
    because: "No HTTPS" },
]

The model's only job is to produce the Signal[] for a site, with evidence attached — which page, which element. Scoring is a fold over that list, and it runs in microseconds with no API call.

What this buys

Replay. Every score can be recomputed from stored signals. Change a weight and re-score 40,000 prospects in a second, without re-crawling anything or spending a penny on inference. That single property is why the rule pack exists — tuning is otherwise a crawl plus an inference bill.

Explanation. Every point traces to a signal, and every signal has a because string and a piece of evidence. A prospect at 55 is not "55 because the model said so"; it is 30 for the broken form plus 25 for no booking, with the URLs. That is what makes the outreach honest — the email cites a specific problem because the score was built from specific problems.

Stability. The weightings do not drift when a provider ships a new model version. Perception may improve; policy stays where we put it.

Testability. A rule pack is a pure function. Its tests are a table of signal sets and expected scores, and they run in CI in milliseconds.

The part that took a rewrite

The first version asked the model for signals and a confidence, then multiplied points by confidence. It seemed reasonable — weight the evidence by how sure we are.

It made scores incomparable. Two prospects both missing online booking would score 25 and 19 because the model was more certain about one page's layout than another's, and that difference had nothing to do with the prospect's value. Sorting by score was partly sorting by how legible their website happened to be.

The fix was to make confidence a gate, not a multiplier:

// A signal is present or it is not. Below the threshold it does not
// count, and it does not partially count either.
const confirmed = signals.filter((s) => s.confidence >= 0.8)
const score = confirmed.reduce((n, s) => n + weightFor(s.kind), 0)

Now equal evidence produces equal scores, and uncertain observations are simply absent rather than quietly deflating a number. Uncertainty is a real thing and the place to represent it is per-signal presence, not smeared across the total.

Where the line generalises

The same split applies well beyond prospect scoring. Anywhere a system reads something messy and then makes a consequential decision:

• Extraction, then validation. The model reads the invoice; arithmetic checks the line items sum.
• Classification, then routing. The model labels the ticket; a rule table decides who gets paged.
• Retrieval, then eligibility. The model finds the policy; code decides whether this customer qualifies.

In each case the model does perception and deterministic code does policy. The test for where the line goes is one question: would you be comfortable explaining this decision to the person it affects, using only what the system recorded? If the honest answer requires "the model weighed it up", the judging half is in the wrong place.

ColdDeck is ours, and it is also our own outbound engine. More in why our crawler is deliberately slow, the case studies, and adding provenance to AI outputs.