Chaos Testing
Introduce unexpected failures or disruptions to chaos test your app with mirrord.
mirrord lets you inject artificial failures and disruptions into your app's outgoing traffic, so you can test how it behaves under unexpected conditions. Create rules that add latency to connections to specific hosts, or turn them into errors, and scope each rule to a single mirrord session.
How chaos rules work
A chaos rule pairs a selector, which picks the traffic to disrupt, with an effect, which defines the disruption. Rules are attached to a single mirrord session and never affect other sessions.
Selectors
A selector matches outgoing connections by their destination:
upstream: the destination host, orhost:portto match a specific port.percentage: roughly how often a matched connection gets the effect (0 to 100).
Currently selectors can only match outgoing TCP connections. Selectors for file operations and HTTP requests are planned.
Effects
An effect defines what happens to a matched connection. Two effects are supported:
latency: delays the connection's read and/or write operations.
{
"name": "latency for api.example.com",
"priority": 0,
"selector": {
"upstream": "api.example.com:443",
"percentage": 100
},
"effect": {
"latency": {
"read_ms": 100,
"write_ms": 200,
"jitter_ms": 25
}
}
}connection_error: fails the connection.typecan be one of:reset(can be applied to ongoing connections),timed_out,refused.
Priority
When multiple rules match the same connection, only one is applied: the rule with the highest priority value. If not set, priority defaults to 0, the lowest.
Managing chaos rules
Option 1: CLI
Prerequisites: Minimum mirrord CLI version 3.241.0
The easiest way to manage chaos rules is with the mirrord chaos commands. To see available commands and options:
All you need to manage rules for a mirrord session is the session ID. When starting a session by running mirrord exec, for example, this is printed by default:
Alternatively, you can run mirrord sessions to show the details of running sessions.
For convenience, you can export the value:
Creating a rule
To add a new rule, for example:
Either write the rule to a file path/to/rule.json and run:
Or, add a new rule straight from stdin:
Multiple rules can be added at once by providing an array of rules instead of just one. Note that the rules are still added individually internally, so if one of the rules is invalid, the others may still be added.
Listing rules
To show the rules for the session:
Modifying a rule
To edit an existing rule, the rule ID is required - this is printed when a rule is added with mirrord chaos add, or can be found in mirrord chaos list:
As with mirrord chaos add, new rules can be provided from a file or stdin. Only one rule can be modified at a time.
Deleting a rule
To remove a specific rule:
To remove all rules for this session:
Formatting
By providing the --format argument, the output of mirrord chaos can be customised:
pretty(default): print a human readable summaryjson: print output in JSONsilent: do not print output (errors are still printed tostderr)
When using mirrord chaos in scripts, it is recommended that you use --format json as the default pretty output may be changed in the future.
Option 2: UI dashboard
For more information on the UI server, see Local UI.
Prerequisites: Minimum mirrord CLI version 3.235.0
In addition to the chaos sub-commands, it is possible to manage and view chaos rules in mirrord's UI server. When running the mirrord chaos command, it starts the UI silently in the background if it is not already running. To start the server and open the dashboard in your browser, run:
The chaos tab on the right hand pane allows you to manage chaos rules interactively.
You can also see traffic for the session, and add rules from here.
Stopping the server
To stop the UI server manually, for example if it is misbehaving, run:
Option 3: REST endpoints
Prerequisites: Minimum mirrord CLI version 3.232.0
Setup: UI server and starting a session
Chaos rules are managed through the mirrord UI server, so make sure it is running:
For more information on the UI server, see Local UI.
Grab the API token and the server address from the output:
Next, start the mirrord session you want to apply chaos rules to. In this example we run it from the terminal, but you can start it from the IDE plugins too:
Grab the session ID from the session output (logs shortened for clarity):
The chaos API needs three values: the server address, the API token that authenticates each request, and the session ID that scopes your rules to this session only. Export them once with your own values, and every example below can be copied as-is:
Managing rules via the API
Creating a rule
To create a chaos rule, write it to a JSON file, for example latency-rule.json:
Then send it in an HTTP POST request to the UI server with the x-auth-token header:
The response returns the created rule, including the id you'll need to modify or delete it later. Note that in responses the effect is nested inside the selector, and upstream comes back with an explicit port, where 0 means any port:
The outgoing traffic (roughly 35% of it) from the service we are debugging with mirrord will now be affected by latency on read operations, when this traffic is destined to a host that matches the selector. In this example, outgoing traffic on this database connection would be impacted by this latency rule, so we can test how the app.js service behaves when there's latency requesting data from this database.
Listing rules
To list all the chaos rules that are active for a session, send an HTTP GET request:
The response is an array of rules in the same shape as the creation response, each with its id.
Modifying a rule
Modifying or deleting a rule requires its ID, which is returned when the rule is created and included in the list response. To modify a rule, edit the rule's JSON file:
Then send it in an HTTP PUT request with the rule ID at the end of the URL:
Here we changed both the name of the chaos rule and its percentage.
Deleting a rule
To delete a chaos rule, send an HTTP DELETE request with the ID of the rule you want:
The chaos rule (in this example 6b8f1c4e-2a73-4d9b-8e56-c3f0a7d1b924) will be deleted, and will stop affecting the session immediately.
Your AI assistant can generate and manage chaos rules for you. Install the mirrord skills plugin and ask it, for example, to "add 750ms latency to my service's database connections".
Last updated
Was this helpful?