Service Chaos engineering services · disaster recovery testing · UK

Chaos engineering services for the team whose DR plan. hasn’t been tested since 2022.

Chaos engineering services for UK SaaS, fintech, and regulated teams. Your backups run nightly, but nobody’s rehearsed the restore. We run the game days you’ve been putting off — then leave you a tested plan and a runbook your on-call follows.

Send a 5-line brief Back to engineering

24hreply, from a senior

100+projects shipped since 2019

Senioronly, on the spine

(Why CTOs sign)

You need chaos engineering services. We ship them with senior engineers and the receipts your acquirer asks for.

Resilience isn’t a dashboard you buy. It’s a thing you rehearse until the on-call rota stops dreading the pager. We run the drills, you keep the tested plan.

Send a 5-line brief

0

Chaos engineering engagements run since 2021/01

0

Production incidents caused on a shipped engagement/02

5day

Fixed-scope audit. £8K. 30-page brief. Six ADRs. Fixed quote./03

The CTO this page is for“backups run nightly” · restore time: unknown

Helena had backups. The backups ran nightly. Nobody had ever restored one.

Helena ran engineering at a UK fintech with 280 customers. The DR plan was a Confluence page last edited in 2022. Backups ran nightly and the dashboard was green. When her board asked “what’s our recovery time if the primary region goes down?”, the honest answer was nobody knew, because nobody had ever tried.

An enterprise prospect sent a security questionnaire asking for the last DR test date and the documented RTO and RPO. There wasn’t one. The deal stalled. The in-house attempt had been two engineers part-time, no game days, no audit log, and a runbook nobody followed.

We ran the first game day in week two. The restore took 9 hours, not the 4 the plan claimed. We fixed it, drilled it again, and left her with a tested plan, a runbook, and an RTO under 4 hours. This page is for the CTO who decided resilience is the product floor and wants it rehearsed by senior engineers, not bought as a tool.

Game days since 2021

The non-negotiables

Resilience as a practice.
Eight things every chaos engagement leaves behind.

Not a one-off stunt. The discipline your on-call follows after we’ve gone. Open any row.

We write down what we expect to happen, then break it. No blind “kill a box and watch”. Steady-state defined, blast radius bounded.

Start in staging, then a single tenant, then a cohort. An abort button on every experiment. Nobody’s production goes down because we ran a drill.

We restore from your real backups into a real environment and time it. RTO and RPO measured, not claimed. A green dashboard isn’t a recovery.

We drop the primary region and watch the failover. DNS, replicas, queues, secrets, the lot. The first time you fail over shouldn’t be the real one.

Latency, errors, and timeouts injected into Stripe, your queue, your database. Circuit breakers and fallbacks proven under real failure, not in a unit test.

Step-by-step recovery, tested during the drill, not written after it. The on-call engineer at 3am reads it and acts. No tribal knowledge, no hero.

A quarterly drill your team runs without us. We teach the muscle, hand over the templates, and watch your engineers run the next one.

Last test date, measured RTO and RPO, drill log, runbook. The exact answers your enterprise prospect’s security questionnaire asks for, in writing.

The eight resilience pains we hear in every audit call

The pain. The day-1 chaos engineering answer.

Each one is the difference between a DR plan you can take to the board and a green dashboard that hides an untested restore.

01
The untested restoreDay-1 architecture
“Our backups run nightly, but I honestly don’t know how long a restore would take.”
Restore drill in week one, RTO and RPO measured. We restore your real backup into a real environment and time it. Documented, rehearsed, on the calendar quarterly. Restore in under 4 hours.
02
The silent failureDay-1 architecture
“A workflow died quietly. The customer found out before we did.”
Per-workflow observability with alerts that page. OpenTelemetry traces, PagerDuty, Slack. We inject the failure during a game day to prove the alert fires before a customer does.
03
The region questionDay-1 architecture
“If our primary AWS region goes down, what happens? Nobody can tell me.”
Regional failover rehearsed, not theorised. We drop the primary region in a drill and watch the failover. DNS, replicas, queues, secrets. The first failover isn’t the real one.
04
The dependency blind spotDay-1 architecture
“Stripe had a wobble and our checkout fell over. We thought we’d handled that.”
Fault injection into every third-party dependency. Latency and errors injected into Stripe, your queue, your database. Circuit breakers and fallbacks proven under real failure with Gremlin and AWS FIS.
05
The bus factor of oneDay-1 architecture
“Only one engineer knows how to bring us back up. If they’re on holiday, we’re exposed.”
Runbook drilled by the whole rota. We run the recovery as a team, write the runbook during the drill, and prove anyone on-call can follow it. Bus factor moves from one to the whole team.
06
The procurement bounceDay-1 architecture
“An enterprise prospect asked for our last DR test date. We didn’t have one.”
Evidence pack ready for the security questionnaire. Last test date, measured RTO and RPO, drill log, runbook, SOC 2 stance. The exact answers procurement asks for, in writing. Deal unblocks in days.
07
The blast-radius fearDay-1 architecture
“I’m nervous about running chaos in production. What if a drill causes the outage?”
Bounded blast radius, abort button, staged rollout. We start in staging, then one tenant, then a cohort. Every experiment has a kill switch. We’ve caused zero production incidents on a shipped engagement.
08
The one-off stuntDay-1 architecture
“We did a chaos day once. It was fun, but nothing changed and we never did it again.”
A repeatable practice, handed to your team. Templates, a quarterly calendar, and a blameless game-day format your engineers run without us. The muscle stays after we’ve gone.

Chaos engineering tech stack · Gremlin + AWS FIS + OpenTelemetry

The tooling we run every chaos engagement on.

Fault injection your team can rerun, observability that proves the experiment, and the AWS-native primitives that scale the practice.

Fault injection + game days

what we break with

GremlinAWS FISChaos MeshLitmusToxiproxyk6 / LocustOpenTelemetryDatadog APMGrafanaPagerDutySentryGitHub Actions

When your brief calls for it

reach when needed

Chaos ToolkitPumbaSteadybitPython (failure scripting)PrometheusVelero (restore)

The infrastructure we drill against

AWS-default

AWSKubernetes (EKS)AWS RDS / AuroraAWS BackupRoute 53 failoverSQS / SNSRedisTerraformDockerCloudWatchAWS KMSVault

Results

Helena’s DR,
after the drills, in numbers

We turned an untested 2022 plan into a rehearsed practice: a timed restore, a regional failover drill, dependency fault injection, a runbook the whole rota follows, and an evidence pack for procurement.

Recovery

9h<4h

Measured restore time

Region failover, rehearsed

Procurement

StalledSigned

Enterprise deal unblocked

Evidence pack, on file

Track record

Chaos engagements since 2021

Incidents caused by a drill

How a chaos engineering engagement starts

Audit, drill, then handover.

The honest sequence. Start with a fixed-price audit, scope the sprint from what we find, and walk away leaving the practice with your team.

01Where everyone starts

5-day audit

Two senior engineers read your estate and your last DR plan, then write it down. £8,000 fixed.

30-page written brief
Six ADRs and a risk matrix
First restore timed
Fixed-price quote for the sprint

02The fixed-scope sprint

Chaos sprint

8 to 14 weeks of game days, fault injection, and failover drills. From £35,000 fixed.

Game days with bounded blast radius
Regional failover rehearsed
Dependency fault injection
Runbook your on-call follows

03Optional, after the sprint

Resilience retainer

One senior engineer, one day a week, for 3 to 6 months. From £5,000 / month.

Quarterly game days run with you
New surfaces drilled as you ship
Observability and alert tuning
Your team takes the wheel

Pricing is plain on the pricing page. Audit £8K. Sprint typically £25-75K fixed. 30-day walk-away both ways.

Chaos engineering services · honest answers

What CTOs actually ask before signing

Pain-first, soft-second.

The audit is £8K fixed: five days, two senior engineers, a 30-page brief, six ADRs, and your first restore timed. The sprint is typically £25-75K fixed-price, scoped from what the audit finds. No day-rate, no scope creep. The plain numbers live on our pricing page.

No, because we never start in production. Every experiment has a written hypothesis, a bounded blast radius, and an abort button. We drill in staging, then a single tenant, then a cohort, and only widen once the failure path behaves. We’ve caused zero production incidents on a shipped chaos engineering engagement.

A backup that runs is not a recovery you can trust. Until you’ve restored one into a real environment and timed it, your RTO is a guess. Our disaster recovery testing restores your real backup, measures how long it actually takes, finds the gap, and rehearses it until the restore is under 4 hours and documented.

Gremlin, AWS FIS, and Chaos Mesh for fault injection, OpenTelemetry and Datadog for the observability that proves the experiment, PagerDuty for paging. We pick what your stack already runs, document it, and leave you the templates so your engineers run the next game day without us. No lock-in, no managed dependency on us.

An evidence pack written for exactly that questionnaire: last test date, measured RTO and RPO, the drill log, the runbook, and a SOC 2 stance. Procurement teams want proof, not promises. Helena’s stalled deal cleared in days once she could send the document instead of an apology.

30-day walk-away both ways, milestone billing in quarters, IP assigns on commit. We’re UK VAT registered, listed on Companies House, and shipping since 2019. You’re never more than a few weeks at risk of paying for nothing.

Every chaos engineering engagement has two senior engineers paired, not one. Every decision goes into an ADR the same day, and the runbook is written as we drill, not after. Two handovers in seven years, both inside 48 hours. The bus factor on your side goes up, not down.

Yes, with 14 days’ notice. Engineers move to other work, spend pauses, and you resume with 14 days’ notice. No cancellation fee. The audit deliverables and any drills run so far stay yours regardless.

Chaos engineering services — product screenshot / UI

In context

The surface you hand over.

chaos engineering services, in context — the dashboards, flows and components your team actually ships, reviews and maintains.

Get a DR plan you can take to your board

One paragraph. That’s it.

Tell us your current estate, when your last DR test was, and the outcome you need. Mohit replies inside 24 hours: a clear yes, a clear no, or the one question that decides it.

Write to mohit@empyrealinfotech.com Replies in 24hBounded blast radiusRestore timed in week one

What happens after the email lands

< 24h
A personal reply.
Yes, no, or the deciding question. Straight to your inbox.
Week 1
First restore timed.
We measure your real RTO and write the brief. Audit deliverables in your hands.
Wk 8+
A tested DR plan.
Game days, failover rehearsed, a runbook your on-call follows, evidence on file.