Monitoring

Monitoring with mirrord for Teams

The mirrord Operator can produce logs in JSON format that can be digested by most popular logging solutions (DataDog, Dynatrace, etc). To enable JSON logging, set operator.jsonLog to true in the Operator Helm chart values. The log level is INFO by default, and can be changed by setting operator.logLevel in the Helm chart values, or alternatively by using the RUST_LOG environment variable in the Operator container, which takes values in the following format: mirrord={log_level} (e.g. mirrord=debug).

This feature is available to users on the Team and Enterprise pricing plans.

Functional Logs

The following logs are written with log level INFO, and can be used for dashboards within monitoring solutions in order to monitor mirrord usage within your organization:

Log messages:

• Copy Target

• Port Steal

• Port Mirror

• Port Release

• Session Start

• Session End

Fields:

field	description	events
target	the session's target	All
client_hostname	whoami::hostname of client	All
client_name	whoami::realname of client	All
client_user	Kubernetes user of client (via k8s RBAC)	All
client_id	unique client id produced from client's certificate	All
session_id	unique id for individual mirrord sessions	Port Steal Port Mirror Port Release Session Start Session End
session_duration	the session's duration in seconds	Session End
port	port number	Port Steal Port Mirror Port Release
http_filter	the client's configured HTTP Filter	Port Steal
scale_down	whether the session's target was scaled down	Copy Target

Prometheus

The mirrord Operator can expose Prometheus metrics if enabled (the default endpoint is :9000/metrics).

Helm

# values.yaml for mirrord-operator helm chart
operator:
  ...
  metrics: true
  ...

Manual

env	description	type	default
OPERATOR_METRICS_ENABLED	enable metrics endpoint	"true" | "false"	"false"
OPERATOR_METRICS_ADDR	metrics http server addr	SocketAddr	"0.0.0.0:9000"

Exposed metrics

metric	description	labels	minimum version
mirrord_license_valid_seconds	Seconds until license expiration		operator 3.101.0 (helm chart 1.15.0)
mirrord_sessions_create_total	Count of created sessions	client_hostname client_name client_user user_id	operator 3.101.0 (helm chart 1.15.0)
mirrord_sessions_duration	Histogram for finished sessions duration	client_hostname client_name client_user user_id	operator 3.101.0 (helm chart 1.15.0)
mirrord_operator_ping_latency	Histogram for round trip latency between the mirrord users and the Operator, helps identify infrastructure issues that may affect mirrord performance	client_hostname client_name client_user user_id	operator 3.122.0 (helm chart 1.35.0)
mirrord_stolen_connections_count	Count of stolen TCP connections	port namespace target user_id	operator 3.122.0 (helm chart 1.35.0)
mirrord_stolen_requests_count	Count of stolen HTTP requests	port namespace target user_id	operator 3.122.0 (helm chart 1.35.0)
mirrord_read_sqs_messages_count	Count of SQS messages read from original_queue	original_queue	operator 3.125.0 (helm chart 1.38.0)
mirrord_sqs_messages_forwarded_to_user_count	Count of SQS messages read from original_queue, forwarded to the local service of k8s_user, local_username.	k8s_user, local_username, original_queue	operator 3.125.0 (helm chart 1.38.0)
mirrord_unmatched_sqs_messages_count	Count of SQS messages read from original_queue that weren't matched by any user's filter and were sent to the main output queue for the deployed application.	original_queue	operator 3.125.0 (helm chart 1.38.0)

OpenTelemetry

The features under the "OpenTelemetry" heading require at least operator chart version 1.46.0.

Exporting Logs

To export logs from the operator to an endpoint, set operator.otelLogExportUrl to the URL in the Operator Helm chart values. You must set this value to export logs. This value does not affect the logs which are printed by the operator to stdout and are always enabled.

The log level is INFO by default, and can be changed by setting operator.otelLogLevel in the Helm chart values (or alternatively by using the OTEL_RUST_LOG environment variable in the Operator container), which takes values in the following format: mirrord={log_level} (e.g. mirrord=debug).

Note that this log level is separate to that defined for logs controlled by operator.logLevel, which are printed by the operator to stdout.

Exporting Traces

To export traces from the operator to an endpoint, set operator.otelTraceExportUrl to the URL in the Operator Helm chart values. You must set this value to export traces.

Context Propagation

This feature requires at least mirrord version 3.184.0.

You can use the mirrord.json file to propagate traceparent and baggage values to the Operator when running mirrord:

{
  "traceparent": "<trace ID>",
  "baggage": "my_key=my_value,another_key=a_second_value"
}

Note that it is expected that the trace ID value of traceparent is not hardcoded, but rather handled and passed in by some wrapper around mirrord. This is because duplicate trace IDs will lead to strange behaviour.

The Operator will propagate these values into exported spans for some frequently used actions, including creating a new resource. Other actions, especially those that result from Kubernetes resource reconciliation, will propagate these values in future versions.

For more info about using traceparent and baggage, see the OpenTelemetry docs about context propagation.

Pre-Built Dashboards

DataDog Dashboard

We offer a DataDog dashboard you can import to track statistics.

Download it here.

Grafana Dashboard

Alternatively there is a Grafana dashboard you can import to track statistics.

Download it here.

fluentd

If you are using fluentd you can add a filter to unpack some values from the "log" message:

<filter kubernetes.var.log.containers.**_mirrord_mirrord-operator-**>
  @type parser
  key_name log
  reserve_data true
  remove_key_name_field true
  <parse>
    @type json
  </parse>
</filter>

This will expand all the extra fields stored in the "log" field.

fluentd + Elasticsearch

Assuming you are using logstash_format true and the connected mapping will store the extra fields in a keyword type, we have a ready made dashboard you can simply import.

Download it here (use Saved Objects to import).