LLM evals that catch a regression before the customer does.
Gold-set curation, eval-runner in CI, drift detection in production. Built on Braintrust, LangSmith, or Promptfoo — whichever fits your stack. £8K audit + fixed build.
What is LLM evals implementation?
LLM evals (evaluations) are automated tests for AI features. They compare the LLM's current output against a curated gold set of known-good answers, so a prompt change or a model bump can't silently break a customer flow. Empyreal Infotech implements LLM evals using Braintrust, LangSmith, or Promptfoo — the eval tool that fits your stack — and wires them into your CI pipeline so every pull request runs the eval suite before merge. Output: a red-line report per PR, drift detection in production, and a gold set that grows with your customers' real questions.
What you get, every engagement.
Gold-set curation
Real customer questions + human-written correct answers. Seeded from your support tickets, grown from production traffic, versioned in git.
CI-integrated eval runner
Every PR runs the eval suite. A red line blocks merge. Cost-per-run monitored so the eval budget doesn't drift.
Drift detection in production
Sampled production traffic scored against the gold set nightly. Alert to Slack when the pass rate drops by more than a threshold.
Model swap without customer damage
Every candidate model runs against the gold set before it goes to 1% of traffic. GPT-5 → Claude 4.6 → whatever's next — the swap is safe.
The LLM evals implementation engagement, week by week.
- 01
Map every LLM call in your product, name the ones customers actually see, pick the eval tool, and design the gold-set schema.
Audit week (£8K). Map every LLM call in your product, name the ones customers actually see, pick the eval tool, and design the gold-set schema.
- 02
Curated from your support tickets + product analytics. Human-written correct answers. First runnable eval in CI by end of week.
First 50 gold examples. Curated from your support tickets + product analytics. Human-written correct answers. First runnable eval in CI by end of week.
- 03
PR bot posts the eval diff. Regression above threshold blocks merge. Cost budget enforced.
CI integration + gates. PR bot posts the eval diff. Regression above threshold blocks merge. Cost budget enforced.
- 04
Sampled traffic scored nightly. Slack alerts on drop. Gold set grows automatically from mis-scored production calls.
Production drift monitoring. Sampled traffic scored nightly. Slack alerts on drop. Gold set grows automatically from mis-scored production calls.
- 05
£3K/month for gold-set curation, eval-tool version bumps, and monthly written health report to the founder.
Optional retainer. £3K/month for gold-set curation, eval-tool version bumps, and monthly written health report to the founder.
Questions we get about LLM evals implementation, with real answers.
Braintrust for teams that want a hosted UI + human-in-the-loop review + strong OpenTelemetry. LangSmith for teams already on LangChain. Promptfoo for teams that want everything local + open-source + git-versioned. We'll recommend based on your stack in the audit week — no vendor bias.
Start with 50 human-written examples covering your top user intents. Grow to 300-500 in the first 90 days by mining production traffic. Anything below 50 is under-powered; anything above 1,000 gets expensive to maintain without paying much marginal accuracy back.
£8K audit + fixed build. Most 2026 builds land at £22-45K depending on the eval tool, the number of LLM surfaces to instrument, and whether you want production drift monitoring on day one. Optional £3K/month retainer for gold-set curation + tool bumps.
Adversarial evals are a separate category — we can seed the gold set with known-bad inputs (Owasp LLM Top-10 patterns, published jailbreak libraries) and score whether your system refuses correctly. Included in the standard build if you flag it in the audit week.
Yes — every eval run captures tokens-in, tokens-out, and $ per call. A model swap that doubles the cost is flagged in the same PR bot comment as an accuracy regression. Cost budgets can be enforced as a merge gate.
The retainer includes a monthly written health report + gold-set audit. Common drift: your product added a new feature and the gold set doesn't cover it. We spot the gap and add examples. Without the retainer you can run the same audit yourself using the framework we hand over.
Send a 5-line brief. That's it.
5 lines: which LLM(s) you use, which product surfaces they power, and any customer-visible bad output in the last 30 days. Mohit replies inside 24 hours.