Helena ran engineering at a UK fintech with 280 customers. The DR plan was a Confluence page last edited in 2022. Backups ran nightly and the dashboard was green. When her board asked “what’s our recovery time if the primary region goes down?”, the honest answer was nobody knew, because nobody had ever tried.
Service Chaos engineering services · disaster recovery testing · UK
Chaos engineering services for the team whose DR plan. hasn’t been tested since 2022.
Chaos engineering services for UK SaaS, fintech, and regulated teams. Your backups run nightly, but nobody’s rehearsed the restore. We run the game days you’ve been putting off — then leave you a tested plan and a runbook your on-call follows.
24hreply, from a senior
200+projects shipped since 2019
Senioronly, on the spine
(Why CTOs sign)
You need chaos engineering services. We ship them with senior engineers and the receipts your acquirer asks for.
Resilience isn’t a dashboard you buy. It’s a thing you rehearse until the on-call rota stops dreading the pager. We run the drills, you keep the tested plan.
0
0
5day
The CTO this page is for“backups run nightly” · restore time: unknown
Helena had backups. The backups ran nightly. Nobody had ever restored one.
An enterprise prospect sent a security questionnaire asking for the last DR test date and the documented RTO and RPO. There wasn’t one. The deal stalled. The in-house attempt had been two engineers part-time, no game days, no audit log, and a runbook nobody followed.
We ran the first game day in week two. The restore took 9 hours, not the 4 the plan claimed. We fixed it, drilled it again, and left her with a tested plan, a runbook, and an RTO under 4 hours. This page is for the CTO who decided resilience is the product floor and wants it rehearsed by senior engineers, not bought as a tool.
Resilience as a practice.
Eight things every chaos engagement leaves behind.
Not a one-off stunt. The discipline your on-call follows after we’ve gone. Open any row.
We write down what we expect to happen, then break it. No blind “kill a box and watch”. Steady-state defined, blast radius bounded.
Start in staging, then a single tenant, then a cohort. An abort button on every experiment. Nobody’s production goes down because we ran a drill.
We restore from your real backups into a real environment and time it. RTO and RPO measured, not claimed. A green dashboard isn’t a recovery.
We drop the primary region and watch the failover. DNS, replicas, queues, secrets, the lot. The first time you fail over shouldn’t be the real one.
Latency, errors, and timeouts injected into Stripe, your queue, your database. Circuit breakers and fallbacks proven under real failure, not in a unit test.
Step-by-step recovery, tested during the drill, not written after it. The on-call engineer at 3am reads it and acts. No tribal knowledge, no hero.
A quarterly drill your team runs without us. We teach the muscle, hand over the templates, and watch your engineers run the next one.
Last test date, measured RTO and RPO, drill log, runbook. The exact answers your enterprise prospect’s security questionnaire asks for, in writing.
The pain. The day-1 chaos engineering answer.
Each one is the difference between a DR plan you can take to the board and a green dashboard that hides an untested restore.
- 01
“Our backups run nightly, but I honestly don’t know how long a restore would take.”
Restore drill in week one, RTO and RPO measured. We restore your real backup into a real environment and time it. Documented, rehearsed, on the calendar quarterly. Restore in under 4 hours.
- 02
“A workflow died quietly. The customer found out before we did.”
Per-workflow observability with alerts that page. OpenTelemetry traces, PagerDuty, Slack. We inject the failure during a game day to prove the alert fires before a customer does.
- 03
“If our primary AWS region goes down, what happens? Nobody can tell me.”
Regional failover rehearsed, not theorised. We drop the primary region in a drill and watch the failover. DNS, replicas, queues, secrets. The first failover isn’t the real one.
- 04
“Stripe had a wobble and our checkout fell over. We thought we’d handled that.”
Fault injection into every third-party dependency. Latency and errors injected into Stripe, your queue, your database. Circuit breakers and fallbacks proven under real failure with Gremlin and AWS FIS.
- 05
“Only one engineer knows how to bring us back up. If they’re on holiday, we’re exposed.”
Runbook drilled by the whole rota. We run the recovery as a team, write the runbook during the drill, and prove anyone on-call can follow it. Bus factor moves from one to the whole team.
- 06
“An enterprise prospect asked for our last DR test date. We didn’t have one.”
Evidence pack ready for the security questionnaire. Last test date, measured RTO and RPO, drill log, runbook, SOC 2 stance. The exact answers procurement asks for, in writing. Deal unblocks in days.
- 07
“I’m nervous about running chaos in production. What if a drill causes the outage?”
Bounded blast radius, abort button, staged rollout. We start in staging, then one tenant, then a cohort. Every experiment has a kill switch. We’ve caused zero production incidents on a shipped engagement.
- 08
“We did a chaos day once. It was fun, but nothing changed and we never did it again.”
A repeatable practice, handed to your team. Templates, a quarterly calendar, and a blameless game-day format your engineers run without us. The muscle stays after we’ve gone.
The tooling we run every chaos engagement on.
Fault injection your team can rerun, observability that proves the experiment, and the AWS-native primitives that scale the practice.
T1
Fault injection + game days
what we break withGremlinAWS FISChaos MeshLitmusToxiproxyk6 / LocustOpenTelemetryDatadog APMGrafanaPagerDutySentryGitHub Actions
T2
When your brief calls for it
reach when neededChaos ToolkitPumbaSteadybitPython (failure scripting)PrometheusVelero (restore)
T3
The infrastructure we drill against
AWS-defaultAWSKubernetes (EKS)AWS RDS / AuroraAWS BackupRoute 53 failoverSQS / SNSRedisTerraformDockerCloudWatchAWS KMSVault
Results
Helena’s DR,
Helena’s DR,
after the drills, in numbers
We turned an untested 2022 plan into a rehearsed practice: a timed restore, a regional failover drill, dependency fault injection, a runbook the whole rota follows, and an evidence pack for procurement.
Recovery
9h<4h
Measured restore time
1
Region failover, rehearsed
Procurement
StalledSigned
Enterprise deal unblocked
1
Evidence pack, on file
Track record
11
Chaos engagements since 2021
0
Incidents caused by a drill
Audit, drill, then handover.
The honest sequence. Start with a fixed-price audit, scope the sprint from what we find, and walk away leaving the practice with your team.
5-day audit
Two senior engineers read your estate and your last DR plan, then write it down. £8,000 fixed.
- 30-page written brief
- Six ADRs and a risk matrix
- First restore timed
- Fixed-price quote for the sprint
Chaos sprint
8 to 14 weeks of game days, fault injection, and failover drills. From £35,000 fixed.
- Game days with bounded blast radius
- Regional failover rehearsed
- Dependency fault injection
- Runbook your on-call follows
Resilience retainer
One senior engineer, one day a week, for 3 to 6 months. From £5,000 / month.
- Quarterly game days run with you
- New surfaces drilled as you ship
- Observability and alert tuning
- Your team takes the wheel
Pricing is plain on the pricing page. Audit £8K. Sprint typically £25-75K fixed. 30-day walk-away both ways.
What CTOs actually ask before signing
Pain-first, soft-second.
The audit is £8K fixed: 5 days, two senior engineers, a 30-page brief, six ADRs, and your first restore timed. The sprint is typically £25-75K fixed-price, scoped from what the audit finds. No day-rate, no scope creep. The plain numbers live on our pricing page.
No, because we never start in production. Every experiment has a written hypothesis, a bounded blast radius, and an abort button. We drill in staging, then a single tenant, then a cohort, and only widen once the failure path behaves. We’ve caused zero production incidents on a shipped chaos engineering engagement.
A backup that runs is not a recovery you can trust. Until you’ve restored one into a real environment and timed it, your RTO is a guess. Our disaster recovery testing restores your real backup, measures how long it actually takes, finds the gap, and rehearses it until the restore is under 4 hours and documented.
Gremlin, AWS FIS, and Chaos Mesh for fault injection, OpenTelemetry and Datadog for the observability that proves the experiment, PagerDuty for paging. We pick what your stack already runs, document it, and leave you the templates so your engineers run the next game day without us. No lock-in, no managed dependency on us.
An evidence pack written for exactly that questionnaire: last test date, measured RTO and RPO, the drill log, the runbook, and a SOC 2 stance. Procurement teams want proof, not promises. Helena’s stalled deal cleared in days once she could send the document instead of an apology.
30-day walk-away both ways, milestone billing in quarters, IP assigns on commit. We’re UK VAT registered, listed on Companies House, and shipping since 2019. You’re never more than a few weeks at risk of paying for nothing.
Every chaos engineering engagement has two senior engineers paired, not one. Every decision goes into an ADR the same day, and the runbook is written as we drill, not after. Two handovers in seven years, both inside 48 hours. The bus factor on your side goes up, not down.
Yes, with 14 days’ notice. Engineers move to other work, spend pauses, and you resume with 14 days’ notice. No cancellation fee. The audit deliverables and any drills run so far stay yours regardless.
In context
The surface you hand over.
chaos engineering services, in context — the dashboards, flows and components your team actually ships, reviews and maintains.
One paragraph. That’s it.
Tell us your current estate, when your last DR test was, and the outcome you need. Mohit replies inside 24 hours: a clear yes, a clear no, or the one question that decides it.
What happens after the email lands
- < 24h
A personal reply.
Yes, no, or the deciding question. Straight to your inbox.
- Week 1
First restore timed.
We measure your real RTO and write the brief. Audit deliverables in your hands.
- Wk 8+
A tested DR plan.
Game days, failover rehearsed, a runbook your on-call follows, evidence on file.