5 Failures Your Service Should Survive (Test Each in 5 Minutes)

With the launch of mirrord Chaos Testing, we wanted to look at some common failure modes that most services usually don’t get tested against, not because they’re unlikely to happen, but because reproducing them on demand has always been annoying enough that people don’t bother. So we came up with a list: five common ways a dependency fails in production, what to check for each, and a small config you can use to test it in about five minutes with mirrord Chaos Testing. Even if you’re not using mirrord (yet), it’s worth learning these failure modes anyway, so you have them in the back of your mind while writing code.

Each failure mode below is accompanied by a mirrord chaos rule you can use to test it. This involves a selector (which outgoing connection to target) paired with an effect (what happens to it). Once you’ve got a rule saved to a file, adding it takes one command after you’re running the service already with mirrord:

$ mirrord exec -f .mirrord/mirrord.json -- node app.js
* session ID: c425f391-e9cc-4199-8de9-7bdbb3e7dfcc

# In another terminal
$ export SESSION_ID='c425f391-e9cc-4199-8de9-7bdbb3e7dfcc'

$ mirrord chaos add -s $SESSION_ID -f rule.json

Swap in the rule for whichever failure you want to test, run your usual request or workflow, and see what actually happens.

1. Your cache is slow but not down #

Most services already handle a cache that’s completely unreachable by having the client throw an error, catching it, and falling back to the database. But a cache that’s slow instead of down is a different failure that’s easy to miss. Without a timeout, a slow cache just makes every request as slow as the cache is, which defeats the whole point of having a cache in the first place. With a timeout, it can be worse (from a latency perspective) than the cache being down outright, since you pay the full timeout wait and then still fall back to the database, instead of failing fast and going straight to the DB.

Cache degradation is rarely total, either. One unhealthy node in a cluster means some connections are slow and the rest are fine, which is exactly why when testing this you should try to target a percentage instead of every connection.

Say you’re running a shopping cart app that looks up a shopper’s session from Redis at checkout, falling back to Postgres on a miss. This is what a rule to test that can look like:

{
  "name": "slow reads from the session cache",
  "selector": {
    "upstream": "redis.cache.svc.cluster.local:6379",
    "percentage": 35
  },
  "effect": {
    "latency": {
      "read_ms": 400,
      "jitter_ms": 100
    }
  }
}

What to check: whether your cache client has a read timeout at all, and whether the fallback you wrote for a dead cache also runs for a slow one.

2. Your dependency is flaky, not fully down #

A dependency that only fails some of the time is often harder to handle than one that’s completely unavailable. The question here isn’t just whether your application survives the failures, but whether a partial problem stays partial, or whether something in your own client turns it into a much bigger one.

Say it’s the search service behind your app’s product search bar. Maybe only some connections to the service are unhealthy while others are working normally. Your application should ideally continue using the healthy connections while dealing with the bad ones gracefully. But retries, connection pools, and circuit breakers can all amplify the original problem if they’re not behaving the way you expect.

You can simulate that by applying connection resets to a percentage of outgoing connections:

{
  "name": "flaky-dependency",
  "selector": {
    "upstream": "search-service.default.svc.cluster.local:8080",
    "percentage": 25
  },
  "effect": {
    "connection_error": {
      "type": "reset",
      "after_ms": 0
    }
  }
}

What to check: does the partial failure stay contained, or does your application amplify it? Watch for retry storms, a connection pool that keeps reusing unhealthy connections, or a circuit breaker that starts rejecting otherwise healthy traffic.

3. Your request failed and the work still happened #

The other failures here are visible from the outside. You send a request, something goes wrong, and you see it in the response or on a dashboard. This one is the opposite: your client reports a clean failure, your error rate ticks up, and the work it was asking for went through anyway.

It happens whenever your client stops waiting before the server stops working. The client’s timeout fires, the request is abandoned, and nothing tells the server that. It finishes the job and commits. Your client, seeing a failure, retries, and the job happens a second time.

Say the same shopping cart app is reserving stock in an inventory service. The reservation succeeds, but slowly enough that your timeout expires first, so the retry reserves the same item again. Your customer gets an error message and the stock is gone twice. Any fault that outlasts your client’s timeout produces this. Latency is simply the easiest one to arrange, so set it above whatever timeout your client uses:

{
  "name": "slower-than-the-client-timeout",
  "selector": {
    "upstream": "inventory-service.default.svc.cluster.local:8080",
    "percentage": 100
  },
  "effect": {
    "latency": { "read_ms": 5000 }
  }
}

What to check: not your client, and not your dashboards. Both will tell you the request failed, and both are wrong. Go count the operations that actually landed on the far side: rows written, reservations taken, charges made. If your client reports one failure and the server did the work twice, the fix is an idempotency key, not a longer timeout.

4. Your dependency slows down under sustained load #

The earlier failures are visible pretty quickly once you trigger them. This one is different because nothing about it shows up if you only send one test request and check the response. It only shows up under sustained, production-like traffic, and by the time it’s visible, it can already be too late.

Say the same shopping cart app publishes an event to RabbitMQ every time a checkout completes, and a separate service consumes those events and writes them to Postgres. Publishing to a queue is usually fast no matter how backed up that queue already is, so checkout itself keeps working and feels completely normal to a shopper the whole time. If the consumer’s database writes slow down, say from 100ms to a full second, it can no longer keep up with a steady stream of new messages, and the backlog just keeps growing. Checkout never notices, because it was never waiting on that write in the first place. That’s what makes this failure dangerous: it’s entirely invisible from the part of the system your users actually touch, and if the queue has no upper bound, it can eventually run out of memory and crash while checkout still looks completely fine.

{
  "name": "slow-dependency-check",
  "selector": {
    "upstream": "slow-postgres.default.svc.cluster.local:5432",
    "percentage": 100
  },
  "effect": {
    "latency": {
      "write_ms": 900
    }
  }
}

What to check: run a load generator against the consumer service for a couple of minutes while this rule is active, publishing at a steady rate, and watch the queue depth in your broker’s dashboard instead of anything checkout itself returns. If queue depth keeps climbing, you’ve found a throughput problem that normal request-level testing wouldn’t have revealed. Now you need to decide how your system should respond: scale or speed up the consumer, bound the backlog, or alert before it reaches a dangerous level.

5. Your dependency is completely down #

This is the failure mode everyone thinks they’ve already handled, since it’s the obvious one: the dependency isn’t reachable, so the call to it fails. What’s actually worth checking isn’t whether that call fails, it’s whether its failure stays contained. A non-critical dependency going down shouldn’t be able to take anything else with it, but that only holds if every call to it is properly isolated, and as applications grow complex, it’s easy to add new code without thinking about that.

For example take a checkout page that shows a few recommendations pulled from a separate, non-critical service. Recommendations have nothing to do with completing a purchase, so a shopper should be able to check out just fine without them. Whether that’s actually true depends on how that call is wired in: an unhandled exception, or the request awaiting it alongside the checkout logic without isolating its failure, can let one optional feature take the whole page down with it.

{
  "name": "dependency-down",
  "selector": {
    "upstream": "recommendations-service.default.svc.cluster.local:8080",
    "percentage": 100
  },
  "effect": {
    "connection_error": {
      "type": "refused",
      "after_ms": 0
    }
  }
}

What to check: does your application still function when a non-critical dependency is unreachable, or does a feature the customer didn’t really need just take down the one they actually came for?

Frequently asked chaos testing questions #

None of this requires much setup #

We used mirrord Chaos Testing for the examples above since it’s what we just shipped, but the underlying point holds regardless of what you use to trigger these faults: all five of these failure modes are worth testing for. What usually stops teams is the friction of setting up an environment to test them.

That’s not the case with mirrord Chaos Testing. It runs these chaos rules on the outbound connections of a service you’re already running locally with mirrord, while that service stays connected to its real dependencies in your existing shared staging cluster. This removes the need for a separate chaos environment or any dedicated infrastructure, since the only thing you need is the mirrord session you’re already running.

Pick the one your own service is least likely to have actually been tested against, and go find out.