How Flash.co achieved +15% quality uplift and −30% cost savings with Divyam.AI's evaluation and routing layer

Two challenges: comprehensive evals and continuous model optimisation

Solving product equality at scale requires getting two things right in sequence, and both are genuinely hard without the right tooling.

Challenge 1: Writing comprehensive, domain-calibrated evals. A small golden dataset of labelled product pairs is a reasonable starting point, but it doesn't generalise. Off-the-shelf LLM judges aren't calibrated to domain-specific nuances: the subtle differences between how a wardrobe is listed on Amazon versus Flipkart require the kind of domain judgment that's difficult to capture in a manually authored prompt. Without a reliable eval, there's no confident way to know whether a change to the model or the prompt is actually an improvement. Building evals that are truly comprehensive is an iterative, labour-intensive process without an intelligent tool to assist.

Challenge 2: Keeping pace with the LLM landscape. Flash was running their product equality task on gemini-2.5-flash-lite, achieving 65.34% task accuracy. Constructing and maintaining a leaderboard across the fast-moving LLM landscape is significant engineering work that competes directly with building the actual product. New models launch regularly; the cost-quality frontier shifts constantly. Without automation, staying current is an ongoing manual investment.

These two challenges compound each other. Without a reliable eval, you can't confidently benchmark alternative models. And without continuous model optimisation, accuracy and cost gains remain on the table. Addressing them requires two things working together: a flywheel that keeps evals sharp, and a routing layer that continuously finds the optimal model.

Two flywheels: one for evaluation, one for optimization

Divyam.AI brought both products to bear on Flash's AI optimization goals: Divyam.AI Evalm8 to sharpen the evaluation layer, and Divyam.AI Router to optimise the model layer. The two work in sequence: a sharp eval is what makes trustworthy model optimisation possible.

Step 1: Build a reliable eval with Divyam.AI Evalm8

Evalm8 begins from a minimal starting point: a two-line seed prompt describing the task: compare two product URLs and determine if they are the same product. Against human annotation, that seed achieved 75% accuracy.

From there, Evalm8 took over. It automatically selected a diverse, coverage-complete set of product pairs for the developer to annotate. That annotation injected domain knowledge directly into the system: the kind of nuanced judgment that distinguishes same-product from similar-product. Evalm8 then automatically evolved the prompt from 2 lines to 40 lines, improving eval accuracy to 80% against human annotation. The developer's bandwidth was spent on annotation, not on prompt engineering.

Step 2: Find the optimal model with the Leaderboard

With a reliable eval in place, Divyam.AI Router automatically constructed a leaderboard, benchmarking nine models against Flash's product equality task. The incumbent model, gemini-2.5-flash-lite, achieved 65.34% accuracy. The leaderboard immediately revealed a better option: switching to gemini-2.5-flash-lite-preview-09-2025 would deliver a +12.33% relative accuracy gain at zero additional cost.

The root cause was instruction-following: Flash's prompt expected a specific JSON schema, and the preview version adhered to it more reliably. This is exactly the kind of ongoing intelligence that an automated leaderboard provides, compounding value without additional engineering investment from the team.

Step 3: Go further with per-request routing

Switching to the best single model from the leaderboard is attractive but sub-optimal. Not all product comparison requests are equally hard. Some pairs are clearly identical; some require careful reasoning across subtle listing differences. A static model choice applies the same level of effort to every request.

Divyam.AI Router makes model selection at a finer granularity: per request, rather than per application. By routing each request to the model best suited to its difficulty and cost profile, the router unlocks gains that no single model can achieve.

Against the gemini-2.5-flash-lite baseline, the router delivered a +15.56% relative accuracy gain, higher than the +12.33% achievable by simply switching models, while simultaneously reducing cost by 11.01%. Better accuracy and lower cost, achieved together.

Better evals. Better models. Better routing. All automated.

The Flash engagement demonstrates what becomes possible when the evaluation layer and the optimisation layer work together. Evalm8 provided the measurement foundation: a robust, domain-calibrated eval that evolved automatically as annotations grew. Router provided the optimisation: a leaderboard and a per-request routing layer that continuously finds the best cost-quality operating point.

Neither product alone would have delivered the full result. A better eval without better routing still leaves accuracy and cost gains on the table. Better routing without a reliable eval means optimising toward a metric you can't fully trust. Together, they form the closed loop that makes production AI systems continuously improve.

All results are from the pilot evaluation on real production data. Both the +15.56% accuracy improvement and the −11.01% cost reduction are measured relative to the original gemini-2.5-flash-lite baseline.

The second upgrade: a smarter router, better results

The Phase 1 pilot established the measurement foundation and demonstrated what intelligent routing could achieve. The Phase 2 pilot upgraded the router itself: a newer version of Divyam.AI Router was deployed against the same model pool, re-benchmarked against Flash's product equality eval, and re-optimised for the cost-quality frontier.

Against the same original gemini-2.5-flash-lite baseline, the Phase 2 router delivered a 15% quality uplift and 30% cost savings. The same models, routed more intelligently, produced a meaningfully better outcome on both dimensions.

This is the compounding dynamic at the core of Divyam.AI's value: the eval stays sharp, the router improves with each upgrade, and the gains compound without additional engineering investment from the Flash team.

A flywheel that keeps turning as the LLM landscape evolves

The LLM landscape is moving fast. New models launch regularly, cost-quality frontiers shift, and the model that was optimal last month may not be optimal today. For a team running a static model assignment, each of these shifts requires a new evaluation cycle, engineering work that competes with building the actual product.

With Divyam.AI's evaluation and routing layer in place, Flash's system is built to improve continuously. As the developer annotates more product pairs, the eval gets sharper. As new models enter the market, the leaderboard updates automatically. As the routing layer sees more requests, it makes better per-request decisions. The developer's bandwidth stays where it belongs: on Flash's product and user experience, not on the infrastructure beneath it.