Evaluation-Driven Development with Divyam.AI
Define quality once. Enforce it on every request.
Evaluation-driven development (EDD) reframes AI work the way test-driven development reframed software: define what good looks like as measurable criteria before you build, score on a 0-100 gradient instead of pass/fail, and iterate on the gap. It is the right practice. But most EDD tooling stops at the notebook, the CI gate, or a dashboard. You measure the gap, then a human still decides. Divyam.AI carries EDD into production. EvalMate turns a few representative examples into a rubric, an aligned judge, and a reward model, and stays the system of record for quality. The Divyam Router then puts that reward model to work: its experimentation layer uses it to benchmark new and fine-tuned models against your bar and train a selector that routes every request to the best-fit model across quality, latency, and cost, while the reward model scores a sample of production traffic, off the critical path, to watch for drift. Evaluation stops being a report you read and becomes a decision your system makes, on every request.
"Does it work?" is the wrong question
Traditional software gives you a green build. You write a test, it passes or it fails, and the answer is the same every time you run it. Agentic and LLM-powered systems do not work that way. The same prompt produces different outputs across runs, quality is a spectrum rather than a boolean, and "does it work?" collapses a hundred distinct questions into one you cannot actually answer.
A practice has emerged to deal with this, and it has a name now: evaluation-driven development. The idea is simple and borrowed directly from test-driven development. Before you write a prompt or pick a model, you write down what good looks like as measurable criteria. Then you measure against it, on a gradient, and you iterate on the gap. It is the most useful reframe in applied AI in the last two years, because it turns a black box into an optimization problem.
Every organization building an agentic application and trying to run it at scale hits the same wall, no matter the domain. How do you know the system is good enough to ship? How do you keep it good as models, prompts, and traffic all change underneath you? EDD is the discipline that answers the first question. This post is about what it takes to answer the second one, in production, automatically.
The EDD loop, the way EvalMate runs it
The EDD cycle has three steps. Each one has a version that is easy to describe and hard to operate at scale, and each is where EvalMate does the work.
Step 1: Define the criteria before you build. This forces a team to articulate what it actually wants: a support agent that is factually accurate 90% of the time, is polite and follows your org's tone and guidelines, and never adopts a defensive tone. Writing that down is the hard part, because domain experts carry the standard in their heads and different reviewers disagree. Instead of asking teams to hand-write assertions, EvalMate turns a handful of representative examples (roughly 100 tagged preferences) into a structured, versioned rubric, then critiques and refines it with expert feedback. The criteria become machine-readable, not tribal knowledge.
Step 2: Measure on a gradient, not pass/fail. A binary check tells you the build is red. A gradient score tells you how far you are from the target: 82 against a 90 threshold is an 8-point gap, and that number is the whole game. EvalMate fine-tunes an aligned judge that scores the way your team would, then distills it into a compact reward model that can score at a fraction of the cost of running a frontier LLM as a judge. The output is not a leaderboard rank. It is a score your systems can act on.
Step 3: Iterate against a baseline. Swap a model, change a prompt, add a retrieval step, and compare the new score against the recorded baseline rather than against your intuition. Prompt engineering stops being an art and becomes a controlled experiment with a measurable result.
A single score is not enough: quality per agent, per dimension
An aggregate score is the most dangerous number in AI development, because it hides exactly the failures you need to find. Agentic applications make this worse: they are built from several agents working together, and one weak agent can drag the whole system while a single number tells you nothing about which one.
EvalMate does not stop at overall quality. It builds a comprehensive measure of quality for every agent in your application. The logic is simple: if an agent has no measurable influence on quality, why is it in the pipeline at all? Measuring each agent tells you exactly where quality is won or lost.
| Agent | Quality score | Threshold | Status |
|---|---|---|---|
| Intent classifier | 96 | 85 | Clears |
| Retrieval agent | 62 | 85 | 23-point gap |
| Answer generator | 91 | 85 | Clears |
| Action and tool agent | 93 | 85 | Clears |
The system-level score was 79, which reads as "mostly fine." The truth is that the retrieval agent is failing and the others are carrying it. That is the agent to fix, and you only see it because EvalMate measures quality per agent instead of blending everything into one number.
It goes one level deeper. Within each agent, quality breaks down by the weighted dimensions of your rubric: correctness, tone and guideline adherence, hallucination control, tool-call correctness, formatting, and whatever else your product cares about. So you learn not just which agent is weak, but which attribute of it: whether the retrieval agent is factually wrong or simply pulling the wrong context. And EvalMate keeps refining the rubric and the aligned judge until every agent and every dimension clears the threshold you define, rather than settling for a one-shot score.
The gap: evaluation that stops at the dashboard
Here is where most of the industry stops. You defined the criteria, you measured on a gradient, you broke the score down per agent and found the retrieval agent sitting 23 points below its bar. The tooling did its job. Now what?
In almost every stack, the answer is: a human reads the dashboard. An engineer files a ticket. Someone runs an A/B test, argues about the result in a review, schedules a migration for next quarter, and hopes the model has not moved by the time the work lands. The evaluation produced a number, and the number produced a meeting. The gap between measuring quality and acting on it is filled with human latency, and that latency is where the cost and the quality both leak away.
Evaluation should not be a report you read. It should be a decision your system makes.
This is the limitation of treating evaluation as a destination. A notebook, a CI gate, and a dashboard are all valuable, and you should have all three. But they are instruments, not actuators. They tell you the gap exists. They do not close it. Quality that lives only in a dashboard is quality you are still enforcing by hand.
Evaluation-driven production: deploy the evals
The step Divyam.AI adds is the obvious one that is genuinely hard to build: take the evaluation you defined and deploy it. EvalMate produces the reward model. The Divyam Router puts it to work in two places, and neither one is a dashboard.
Experimentation trains the selector. The Router's experimentation layer uses EvalMate's reward model to analyze the capabilities of any candidate model against your quality bar: a newly launched frontier model, a new version of your fine-tune, or an open-source model hosted on your own GPUs. From that analysis it trains the Router's selector, the component that decides, per request, which model is most capable for the job and directs the request there. Because the selector is trained offline on the reward model's verdicts, the decision at request time is fast and cheap: no reward model in the hot path, and the Router simply logs each request as it goes.
The reward model watches production. In parallel, off the critical path, the Router scores a sample of that logged traffic with the reward model. Sampling is enough; not every request needs checking. This continuously watches the quality of your production agents, catches drift and regressions, and feeds the results back into EvalMate's system of record and the next round of experimentation.
What you get is an inference infrastructure that keeps upgrading itself: a drastic reduction in cost and latency, and a stack that is always on the best-fit set of models for your quality bar, with no one running a migration.
The loop closes on itself:
- EvalMate turns your examples into a rubric, an aligned judge, and a reward model, and keeps the system of record.
- The Router's experimentation layer uses that reward model to benchmark every candidate model (new releases, your fine-tunes, open-source models on your GPUs) against your quality bar, and trains the selector.
- The selector routes each production request to the most capable model for that request, in real time.
- The Router logs each request and scores a sample of production traffic with the reward model, off the critical path, and drift and coverage gaps flow back into EvalMate's system of record.
- When a better model appears or an agent slips below its bar, experimentation retrains the selector, and the loop repeats.
Nobody runs a migration. The system does. Evaluation-driven development becomes evaluation-driven production.
What deploying your evals unlocks
Once evaluation drives routing instead of just filling a dashboard, a set of outcomes that used to require projects becomes automatic.
- Right-size every request. Simple requests go to the smallest model that clears the bar; only the hardest ones reach for a frontier model. Because the selector is trained on the reward model's verdicts, that decision is made per request and cheaply, and cost drops drastically without touching quality.
- Use open-source models on your own GPUs. A self-hosted open-weights model that clears the bar for a class of requests earns that traffic at a fraction of the API cost, and your data never leaves your walls.
- Every model earns its traffic. Proprietary, open-source, and locally hosted models compete on your quality bar, not on a public leaderboard.
- Never chase an upgrade. New models are benchmarked and adopted automatically the day they ship, so your team never has to stop and run a migration to stay current.
The result is a self-optimizing stack that keeps your quality bar rising while your cost falls, and that is never hostage to the pricing power of a single provider.
In practice
Return to the retrieval agent scoring 62 against an 85 threshold. In a dashboard-only world, that finding starts a workstream: pick a stronger model for retrieval, re-test the whole system, weigh the added cost, ship the change weeks later.
With evaluation deployed into the Router, the same finding is a routing decision. Experimentation has already benchmarked the alternatives against your criteria, so the selector routes the retrieval agent's calls to the model that clears the bar for that step, which might be a domain fine-tune or a larger frontier model, while the intent, answer, and action agents stay on the cheaper models that already clear it. The 23-point gap closes in production, on the one agent that needed it, without raising cost on the agents that were already fine. When next month's open-weights release clears the retrieval bar at a lower price, experimentation flags it and retrains the selector, without anyone scheduling anything.
The evaluation did not just describe the problem. It fixed it.
From development to production
Evaluation-driven development is the right discipline: define quality as measurable criteria, score on a gradient, iterate on the gap, and break quality down per agent and per dimension so aggregate numbers cannot hide the failures that matter. Divyam.AI's contribution is to refuse to let that discipline end at the dashboard. Define your quality once in EvalMate, and the Router enforces it on every request, continuously selecting the best-fit ensemble of models and closing the gap in production rather than in a meeting.
You focus on your product. Divyam.AI keeps you on the highest quality at the lowest cost. Book a demo to run it on your own traffic.
Further reading. For a deeper look at evaluation-driven development as an engineering practice, see Red Hat's write-up on evaluation-driven development with EvalHub, which frames the define, measure, and iterate loop we build on here.