AI Evals as a Service: How I Charge $3,000 to Stop Client AI Features Breaking

I did not plan to sell AI evaluations. I fell into it because a client's support assistant started confidently telling customers that a discontinued product was still in stock, and nobody noticed for eleven days. The feature had shipped, the demo had gone well, and after that everyone assumed silence meant success.

That cleanup job turned into the most repeatable offer I have ever sold. Nine engagements later, the shape is fixed: a $3,000 fixed-scope eval build, then $600 a month to keep it running and report on it. This article is the whole thing — the process, the deliverables, the pricing, what I refuse to promise, and the numbers from the engagements I have finished so far.

What "evals" actually means to a paying client

Strip out the research-lab vocabulary. An eval is a saved test case with an expected outcome, run automatically against a client's AI feature every time something changes. Change the model, the prompt, the retrieval index, the tool definitions — the suite runs and tells you what got worse.

Clients do not buy "evaluation methodology". They buy an answer to one question they cannot currently answer: if we change this prompt on Friday, will we know by Monday whether we broke something? Right now the answer is no. Their entire quality process is a product manager typing four questions into a staging chat and eyeballing the replies.

The three failure types I keep finding

Every engagement surfaces the same categories, in roughly the same proportions.

Silent regressions. Somebody improves the system prompt to fix one complaint and quietly degrades six other behaviours. Without a suite, this is invisible until a customer complains.

Confident fabrication. The assistant answers questions that are not covered by the retrieved documents instead of saying it does not know. This is the one that generates refund requests and angry emails.

Format drift. The feature is supposed to return structured output for a downstream system, and in maybe two percent of cases it returns prose, an extra field, or a slightly different enum value. Two percent is small enough to survive a demo and large enough to corrupt a database.

The four-week build I sell for $3,000

Fixed scope, fixed price, fixed calendar. Clients relax when the deliverable list is boring and the dates are real.

Week 1 — Harvest reality, not imagination

I do not invent test cases. I mine them. I ask for read access to whatever exists: chat transcripts, support tickets, the internal Slack channel where people paste screenshots of bad answers, and the analytics events around the feature.

From that pile I extract 120 to 200 real inputs and sort them into four buckets:

• Core happy path — the questions the feature exists to answer. Usually 40% of the set.
• Edge and adversarial — ambiguous phrasing, multiple products in one question, prompt-injection attempts, off-topic requests. Roughly 30%.
• Must-refuse — legal, medical, pricing-promise and competitor-comparison questions where the correct behaviour is a graceful decline or a handoff. Around 20%.
• Known past failures — every bad answer anyone has ever screenshotted. About 10%, and the most politically valuable bucket, because those are the cases stakeholders remember.

Then the unglamorous part: sitting with a subject-matter expert for two ninety-minute sessions to write the expected outcome for each case. Not the ideal wording — the acceptance criteria. "Must state the 30-day window", "must not name a competitor", "must include the handoff link", "must return valid JSON with exactly these four keys".

This is where the value is created, and it is why an eval suite cannot be bought off a shelf. The criteria encode the client's business rules.

Week 2 — Build the harness

The harness is deliberately plain. A repository with the cases in version-controlled files, a runner that calls the client's feature through the same interface their app uses, and three kinds of checks:

Deterministic checks first. Does the JSON parse? Is the required phrase present? Is the forbidden phrase absent? Is the cited document ID actually in the retrieved set? These are free, instant and never flaky, and they catch more real bugs than people expect.

Model-graded checks second, and only where necessary. For open-ended answers I use a grader model with a narrow rubric and a three-way verdict: pass, fail, needs human review. A vague rubric produces a grader that agrees with itself and nothing else.

Human spot-checks third. Twenty sampled cases per run reviewed by a person, which is also how I measure whether the grader is trustworthy.

The step everyone skips: grade the grader

Before I let a model-graded score into a report, I hand-label 60 cases and measure agreement with the grader. Below about 85% agreement, the rubric gets rewritten. I have thrown away two rubrics entirely because they could not beat a keyword check.

I put that agreement number in every report. It is the single fastest way to earn a technical client's trust, because it shows I know my own instrument's error bars.

Week 3 — Wire it into the workflow

An eval suite that someone has to remember to run is a document, not a safety net. So it runs on every pull request that touches prompts, retrieval or model configuration, and on a nightly schedule against production configuration.

Results land in three places: a pass-rate summary comment on the pull request, a Slack message when a bucket drops more than three percentage points, and a dashboard with the trend line per bucket. That dashboard is what gets screenshotted in board meetings, which is why it exists.

I also set a release gate with the client's agreement: must-refuse cases at 100%, format checks at 100%, core happy path above an agreed floor, usually 92%. Deploy blocked below that. Negotiating those thresholds is a business conversation, not a technical one, and I make the client own the numbers.

Week 4 — Hand over and train

Two recorded sessions: how to add a case in under five minutes, and how to read a failing run. Then a written runbook with the three things I know they will need — how to quarantine a flaky case, how to re-baseline after an intentional behaviour change, and how to add a new bucket when they ship a new feature.

The engagement ends with a live drill. I deliberately break the system prompt, we watch the suite catch it, and the client sees their new safety net do its job. Nothing sells the retainer better.

The tool stack, and why it is deliberately boring

What I use

A plain Git repository holds the cases as YAML files, one file per bucket, so a subject-matter expert can edit them in a browser without learning anything new. The runner is a few hundred lines of TypeScript calling the client's own endpoint — never the model directly, because testing the model instead of the product tests the wrong thing. Scheduled runs and pull-request runs both go through the client's existing CI, so there is no new system for anyone to maintain. Results go to a Postgres table, and the dashboard is a handful of queries.

What I avoid

I stay away from heavyweight evaluation platforms on the first engagement. They are not bad products, but they add a vendor, a seat cost and a lock-in argument to a project whose whole purpose is to build internal confidence. If a client later outgrows the plain setup, migrating a folder of YAML cases into a platform is a one-day job — and by then they know exactly what they need from it.

One rule about flaky cases

A case that passes and fails at random is worse than no case, because it teaches the team to ignore red. When a case flakes twice, it goes into quarantine the same day, gets a note explaining why, and either becomes deterministic or gets deleted within the week. I report the quarantine count in every monthly summary; a growing quarantine list is a symptom, not an annoyance.

Pricing, and what it actually costs me

The rate card as of this month:

Item	Price	What it covers
Eval readiness review	$500	Half-day audit, 20-case starter set, written findings
Eval suite build	$3,000	The four-week engagement above, one AI feature
Additional feature	$1,600	Second feature reusing the harness
Monthly operation	$600/mo	Runs, triage, monthly report, up to 15 new cases
Incident deep-dive	$850	Root-cause analysis of a specific production failure

My direct costs on a build are small and unromantic: roughly $70 to $130 in model spend across development runs and grader calls, plus about $25 a month in cloud compute for scheduled runs. The real cost is my time — 22 to 30 hours on a build, most of it in the criteria sessions and the grader calibration.

The $500 readiness review exists purely as a de-risking step, and it converts better than any proposal I have written. Six of my nine builds started as a readiness review. It is priced to be an easy yes and scoped so I can finish it in one afternoon.

Why I sell the retainer, not just the build

A one-off suite decays. Models get deprecated, prompts get edited by whoever is closest to the complaint, new features appear. The retainer is honest work: I run the suite, triage failures into "real regression" or "case needs updating", add cases from the month's support tickets, and send a one-page report.

Six of nine clients are on the retainer. Two do it themselves, which is a legitimate outcome and I say so during handover. One churned after four months because their AI feature was retired.

The results from nine engagements

I am going to be careful here, because this is exactly the kind of claim that gets inflated in blog posts.

What I can measure directly: across nine suites, the first full run failed on 19% to 43% of cases, median 27%. Every single client was surprised by that number, because their internal spot-checking had suggested near-perfect behaviour. The must-refuse bucket was the worst performer in seven of nine cases.

What I can measure with the client's help: at three clients with clean before-and-after ticket tagging, AI-related support escalations fell by 34%, 41% and 12% over the two months after the first remediation round. The 12% client had a much better baseline to begin with.

What I cannot claim: that evals caused all of that. Those teams were also fixing their retrieval and rewriting prompts, partly because the suite told them where to look. The suite gave them a map. It did not do the walking. I say that sentence out loud in every sales call, and it has never once cost me a deal.

Revenue reality, plainly: two builds and six retainers in the current month is $9,600. Some months are one build and the same retainers. It is a good business, not a passive one, and it is entirely dependent on a client having shipped an AI feature that matters to them.

How I actually find these clients

There is no clever funnel. Three things work.

Free public failure reports. I use a company's public AI feature, run 20 questions through it, and send the founder a short document with three real bad answers, verbatim, plus what a suite would have caught. No pitch in the first email. Response rate hovers around 20%, which is absurd for cold outreach, and it works because the evidence is undeniable and specific to them.

Being present where the pain is discussed. When someone posts "our AI assistant hallucinated at a customer and we don't know why", a genuinely useful reply — the actual first three checks I would run — gets more inbound than any content calendar I have tried.

Referrals from AI implementation shops. Agencies that build these features often do not want to own quality forever. I have a quiet arrangement with two of them: they introduce me at handover, I do not compete for their build work.

Who I turn down

Prototypes with no users, features nobody in the company would notice if they vanished, and any client who wants a certificate that says their AI is accurate. I do not sell certainty. I sell a repeatable measurement and a tripwire, and if the buyer wants a guarantee, the engagement will end badly.

A 30-day start if you want to sell this

Days 1–7. Pick a niche where wrong answers cost real money: fintech support, healthcare intake, legal document assistants, ecommerce policy bots. Learn that niche's must-refuse rules properly, because that is the bucket clients cannot write themselves.

Days 8–14. Build one reference suite against a public AI feature. Write the runner, the deterministic checks, one calibrated rubric. Measure your grader agreement so you have a real number to talk about.

Days 15–21. Turn it into a two-page report template: pass rate by bucket, the five worst failures verbatim, the fix for each, the grader agreement figure. That template is your product demo.

Days 22–30. Send ten free failure reports. Offer the $500 readiness review to anyone who replies. Expect one to convert, and expect that one to teach you more than the previous three weeks.

The unglamorous truth about this offer is that the technology is the easy half. Anyone can write a test runner. What clients pay for is somebody willing to sit with a subject-matter expert and turn fuzzy business rules into 180 concrete pass-or-fail criteria — and then keep those criteria alive month after month while everyone else is busy shipping features.

Nothing here is financial advice. Rates, model costs and demand vary by market, and my numbers are what I invoiced, not what you should expect.