Chaos Testing When Your Testers Are Agents

When you chaos test a dependency in staging, you break it for everyone. So you announce it, you do it in a window, and you put it back.

That is how the usual tooling works. Whether the fault is injected cluster-wide or narrowed by a service mesh to a single header, it is still a rule you create in the cluster, that someone has to approve, and that stays there until it is taken away.

So fault injection has mostly been one tester at a time, with permission. That is acceptable when the testers are people, because people do it rarely. The story changes when the testers are agents, you have more code to test (and more capacity to test it), and you want to verify it more thoroughly.

Nobody runs the whole matrix #

We have a small demo app: an orders service summarizes a customer’s open orders, which it reads from Postgres. It then asks a pricing service what each item is worth, and gets a short-lived credential from a token service to make that call.

Testing that one endpoint against those dependencies is a matrix. Three dependencies, times three ways each can misbehave (slow, refusing connections, gone entirely), times two traffic patterns (one request at a time, and many at once): 18 runs.

Running it is slow. A 250ms lag on the pricing call becomes eleven seconds for a single request once it fans out across an order’s line items. And some of these failures are triggered by time passing rather than by anything you send, so the only way to reach them is to wait: the token cache has to expire, which takes 30 seconds, and a pooled connection has to sit unused long enough to get dropped. A thirteen-case version of this, smaller than the full matrix, spent 166 seconds of every pass doing nothing but waiting and took just under eight minutes to run. Comparing behavior before and after a change needs two of them.

So the full matrix for one endpoint is somewhere around twenty minutes, nearly all of it waiting. That is fine once. It is not fine across the twenty services a real team owns, run one after another, because the whole thing is serialized on a cluster that only one person can be breaking at a time.

Which is why the matrix does not get run. People pick the three scenarios they think matter and skip the other fifteen.

What makes more than one possible #

A mirrord chaos rule is attached to a session, not to the cluster. The fault applies to the outgoing connections of one process, nothing else sees it, it goes away when the session does, and nobody had to approve it.

Here is that measured. Two agents each run their own copy of the orders service against the same deployment at the same time, and one chaos rule puts three seconds of latency on the pricing service for agent A alone. Every request returned 200, so nothing failed anywhere. The only question is whose response time moved.

Agent A is three seconds slower, which is what it asked for. Agent B, working against the same deployment a few milliseconds away, is untouched, and so is the service everyone else in the cluster is talking to. The rule reports one hit, on A’s session, and B has no rules attached at all.

Now the twenty services stop queueing. Twenty agents work at once, against the staging cluster the team is still using, and the whole sweep finishes in the time one service used to take.

What this changes #

Resilience testing today is an event. Someone schedules it, tells people, runs it, and writes it up. It happens rarely, because it costs coordination every time.

Two things change once the coordination is gone. It can run often, on every code change, the way tests already do. And it can run completely - all eighteen runs.

Try it #

The orders service above is at metalbear-co/orders-demo-app. Install the mirrord skills plugin and an agent can create and remove chaos rules the same way it runs any other command:

/plugin marketplace add metalbear-co/skills

Then give one agent something like this:

Work through the resilience matrix for /customers/:id/summary. For each dependency it calls, try latency, connection resets, and total unavailability, under both sequential and concurrent load. Tell me what breaks and what you had to do to make it break.

Then start a second agent on the same service, using the second config the repo ships, and point it at a different dependency. Neither will notice the other, and neither will disturb the cluster.

The chaos testing docs have the full rule schema. Chaos rules work on the open source tier, and you need mirrord 3.250.0 or newer.

If you have not seen the feature itself, Introducing mirrord Chaos Testing covers everything a rule can express, and How To Get Started With Chaos Testing Using mirrord walks a person through the loop by hand.