Service
RAG that is
not vibe-checked by asking it three questions.
Almost everyone has shipped a retrieval chatbot by now. Almost nobody can tell you its recall, whether last week’s prompt change made it worse, or which of the four layers is responsible when it answers wrongly.
That is not a modelling problem. It is a measurement problem, and it is why “we improved the prompt” is the only available lever in most teams.
We build the harness first, then fix what it shows. Usually the harness alone changes the conversation, because it turns an argument about vibes into a number that moved.
The question nobody can answer
“Did that change make it better, or does it just feel better?”
Which layer is actually wrong
“The AI is hallucinating” is almost never the diagnosis. It is a symptom of one of these, and they need different fixes.
The evaluation harness
The deliverable that matters more than any single fix. It is what lets your own team keep improving the system after we leave.
The golden set
Fifty to two hundred real questions with known-correct answers and the passage each answer should come from. Built from your actual support tickets, not invented.
Retrieval metrics
Measured separately from generation. Recall@k tells you whether the right passage was even a candidate. Most "the LLM is wrong" bugs are actually recall failures.
Answer grading
Faithfulness to the retrieved context, and correctness against the known answer. Two different failures that need two different fixes.
Regression gate
It runs in CI. A prompt change that drops recall by 6% fails the build instead of shipping and being discovered by a customer.
Abstention
The system has to be able to say it does not know. A RAG system that always answers is a RAG system that sometimes lies.
Trace
Every answer stores which chunks it used and what they scored, so "why did it say that" has an answer that is not a shrug.
An honest note on this page
“RAG consultant” is the most crowded phrase we compete on, and a lot of people offering it have built one demo.
Our retrieval work predates the current wave, and the thing we bring is not a novel architecture — it is the discipline of measuring before changing. If you already have a golden set and a regression gate, you genuinely do not need us. Hire an engineer.