Introducing mirrord Chaos Testing

Most teams, when writing code for microservices-based applications, only ever test it against dependencies that happen to be healthy at the time of testing. That’s the happy path. The real test is what happens when they’re not, and that’s usually only discovered once it’s already become an incident.

We’ve launched mirrord Chaos Testing, which lets you deliberately break connections to your dependencies on demand and see how your code actually responds. In this post, we’ll cover why teams need this, how chaos rules work, how it’s different from other chaos engineering solutions, and how to get started.

Why Most Teams Only Test the Happy Path #

Distributed systems fail in ways that are difficult to reproduce on demand: a database gets slow under load, a downstream API starts timing out intermittently instead of going fully down, a connection resets mid-request because a pod got rescheduled. Every team has a war story like this, and almost none of them tested for it beforehand, because simulating realistic failures is hard. You can fake them locally by mocking a dependency or adding sleeps and errors to your code, but that only approximates how the real system behaves. Or you can use a dedicated chaos engineering platform against a real environment, which gives you more realistic failures but comes with significantly more setup and coordination.

Tools like Chaos Mesh and LitmusChaos are useful for the latter, but they’re built for a different moment in the development cycle. They inject faults into the shared environment, so the effect lands on everyone using it, not just you, and you need to coordinate before you can run one. That makes sense for testing a final build before shipping it, but it’s the wrong tool for a developer who just wants to know, right now, whether the retry logic they wrote actually works before they open a pull request. Setting up a dedicated chaos testing environment for that kind of everyday check is rarely worth the effort, so most teams just skip it and hope their error handling was written correctly the first time.

mirrord already solves a version of this problem. It connects a service running outside the cluster, on your machine, in CI, or in an agent’s sandbox, to your staging cluster, so its traffic, files, and environment variables come from the real thing and your code behaves as if it were running there. Chaos Testing extends that same idea one step further. Instead of just watching how your service behaves with healthy dependencies, you can now simulate slow or unreachable dependencies for your own service’s connections and see how it responds, without affecting the actual dependency or anyone else using it.

What Is mirrord Chaos Testing #

A chaos rule pairs a selector, which picks which of your running service’s outgoing connections to disrupt, with an effect, which defines what happens to them. Since these rules attach to the mirrord session for the service you’re running locally, they only ever affect your own traffic, never anyone else’s.

Selectors match by destination (a host or a host:port if you want to be more specific) plus a percentage for how often the effect applies. Percentage is important because a dependency that’s fully down is the easy case to handle, but a dependency that fails intermittently, say 30% of the time, is a much more realistic (and much easier to miss) failure mode.

Two effects are currently supported:

• latency delays a connection’s reads and/or writes by a set number of milliseconds, with optional jitter.
• connection_error fails the connection outright, either as a reset (which can be applied mid-connection), a timed_out, or a refused.

Say your service reads a user’s session from Redis and falls back to Postgres when Redis is unavailable. A dead Redis instance is straightforward: the client returns an error, your fallback runs, and you read from Postgres instead. A slow Redis instance is more interesting.

Instead of failing fast, your service might spend hundreds of milliseconds waiting for the cache before eventually falling back to Postgres anyway. And cache degradation is rarely all-or-nothing, so one unhealthy node might make some connections slow while the rest behave normally. Here’s a rule that gives 35% of matching Redis reads an extra 400ms of latency, with up to 100ms of jitter:

// ./chaos-rules/01-slow-cache.json

{
  "name": "slow reads from the session cache",
  "effect": {
    "latency": {
      "read_ms": 400,
      "jitter_ms": 100
    }
  },
  "selector": {
    "upstream": "redis.cache.svc.cluster.local:6379",
    "percentage": 35
  }
}

With this active, you can find out whether your Redis client has a read timeout at all, and whether the fallback path you wrote for a dead cache also behaves correctly when the cache is merely slow. You can also see what happens when only some reads are slow rather than Redis disappearing entirely.

One thing to keep in mind when choosing the latency is that mirrord applies it per read, not once per application request. If handling a request involves multiple reads from the connection, the added latency can accumulate. So choose the value based on the degraded behavior you want to simulate rather than simply working backwards from your application’s timeout.

How to Get Started #

Chaos Testing works with any mirrord session, including the mirrord OSS. No operator or mirrord for Teams license required (chaos rules need mirrord CLI 3.241.0 or later). The most direct way to add a chaos rule is via the mirrord chaos subcommands. First, you need an active mirrord session:

mirrord exec -f mirrord.json -- npm run dev

This prints a session ID, which every mirrord chaos command needs:

export SESSION_ID='c425f391-e9cc-4199-8de9-7bdbb3e7dfcc'

mirrord chaos add -s $SESSION_ID -f path/to/rule.json
mirrord chaos list -s $SESSION_ID
mirrord chaos delete -s $SESSION_ID

If you’d rather not manage JSON files by hand, running mirrord chaos also starts a local UI server in the background (or start it yourself with mirrord ui), which gives you a dashboard for adding and reviewing rules, plus a live view of the traffic during your session.

And if you want to script this into CI or your own tooling, the same UI server exposes REST endpoints for creating, listing, updating, and deleting rules over HTTP.

There’s also a more direct route if you’re working with an AI coding agent like Claude, Cursor, or Codex: with the mirrord skills plugin installed, you can just ask your AI coding agent to “add 750ms latency to my service’s database connections” and it’ll generate and manage the chaos rule for you.

Stop Finding Out How Your App Handles Failure in Production #

The harder question isn’t whether your error-handling code works in isolation, but whether your service behaves the way you expect when talking to real dependencies and one of them becomes slow, flaky, or partially unavailable. Chaos Testing lets you test those conditions against the same dependencies your service normally talks to, without requiring a dedicated environment or disrupting anyone else using the shared cluster. You run your service with access to all of its real dependencies in the cluster, then inject the fault into one of them at the level of your session only, see how your service behaves, and fix problems before they show up in production.

Read the full docs to see everything chaos rules can do, and drop into our community Slack if you have any feedback for us after trying it out!