> For the complete documentation index, see [llms.txt](https://metalbear.com/mirrord/docs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://metalbear.com/mirrord/docs/changelog/operator.md). # Operator The release changelog for the mirrord operator. ### 3.236.0 - 2026-11-20 #### Security * Moved the `operator-kafka-proxy` image to the Debian 13 distroless base and raised its `netty` and `jackson-databind` dependencies to fixed releases, closing the critical `libbrotli1`, `netty-handler` and `jackson-databind` vulnerabilities that image scans reported. The JVM truststore now comes from the Debian `ca-certificates` package instead of the Temurin bundled store, which drops six roots the previous image trusted (OISTE Server Root ECC G1 and RSA G1, SwissSign RSA TLS Root CA 2022 - 1, TrustAsia TLS ECC and RSA Root CA, emSign ECC Root CA - G3) and re-adds ten it did not (Baltimore CyberTrust Root, GlobalSign Root CA, Entrust.net Certification Authority (2048), Go Daddy Class 2, Starfield Class 2, XRamp Global CA and the four CommScope Public Trust roots), so brokers whose certificates chain only to one of the dropped roots, or that want to pin specific roots, need an explicit `ssl.truststore.*` configuration. #### Added * The operator logs, for each target pod of a session, when an agent loads, fails to load or is lost, and each port that the agent subscribes. #### Changed * A preview session's operator log lines name the preview's key as well as its target. * Bumped the bundled Cloud SQL Auth Proxy to 2.28.0. #### Fixed * Branches of ClickHouse Cloud databases now start: their `Shared*MergeTree` tables get the matching plain `*MergeTree` engine. * ClickHouse branches now copy column names and comments that contain text like `ReplicatedMergeTree('…', '…')` as they are, instead of rewriting them along with the table engine. * ClickHouse branches now copy rows when the source user's settings profile sets `asterisk_include_materialized_columns` or `asterisk_include_alias_columns`. * ClickHouse branches now copy the rows of tables whose column names contain a backtick or a backslash. * ClickHouse branches now start when a copied table has no rows. * Database branch init containers retry a source connection that fails for a moment instead of failing the branch. * Liquibase branch migrations fail fast on a changelog that cannot apply. ### 3.235.0 - 2026-11-18 #### Security * Preview environments no longer write the values of credential headers, such as `Authorization`, `Cookie` and API key headers, to the operator's logs when they receive an HTTP request. With `operator.hideHeadersAndProperties` set to `true`, these log lines are left out entirely. #### Infrastructure * Added SSO setup to the Helm chart: an optional Ingress exposes the operator API to OIDC logins at a public URL #### Changed * Connect failures log the target, CLI version, session mode and multi-cluster choice, and client-side refusals log at info. #### Fixed * A filtered PostgreSQL table whose last row ends in an empty value no longer fails the branch with "missing data for column", and trailing spaces in the last value are kept. * Documented that `mssqlBranching` accepts the SQL Server license terms. * Filtered PostgreSQL copies of tables with generated columns no longer fail. * Fixed RabbitMQ splitting needlessly requeueing messages held for a missing session queue. * Outgoing connections to cloud metadata and identity endpoints, such as the instance metadata service at `169.254.169.254`, and to other link-local addresses now always leave from the same pod when targeting a workload with multiple pods. Node instance role credentials obtained through AWS IMDSv2 now work in such sessions. * PostgreSQL branches from sources on 14.25, 15.20, 16.16, 17.12, 18.7 or later now come up on branch images from before those releases. ### 3.234.0 - 2026-11-17 #### Added * Preview environments can now run without a target to provision branch databases alone: the session registers on its branches and keeps them alive for its TTL, running no pod. A branch pod takes its connection values from literals, Kubernetes Secrets, or ConfigMaps named directly, since there is no target pod to read them from or to borrow a cloud identity from; S3 and turbopuffer branches, which run no pod, keep their secret manager sources too, and Spanner branches still need a target. #### Changed * A `MirrordSplitConfig` with BullMQ queues now needs `{{RANDOM}}` in its `tmpNameTemplate`, as it already does for RabbitMQ, SQS and Kafka queues, so that two splits of the same queue never get the same temporary queue names. A config whose template lacks it is rejected. Before you upgrade, end the sessions on such a config and let its split end: the upgraded operator can't resume a split that runs on such a template, and leaves the Redis keys of its main queue behind. * To keep the messages left in a GCP Pub/Sub split's temporary subscription when the split ends, the operator needs `pubsub.subscriptions.get` and `pubsub.subscriptions.consume` on the temporary subscriptions it creates. Without them, it deletes the subscription together with those messages, as before. #### Fixed * A Temporal task that matches a session's filter just as the session ends now goes to the deployed worker. Before, it waited in a virtual task queue that no worker polled, until Temporal timed it out. * A Temporal worker now receives a task right away even when another worker on the same virtual task queue closed its poll earlier, for example because its pod restarted. Before, the task could wait up to 20 seconds. * Deleting a `MirrordSQSSession` or `MirrordRMQSession` that an older operator left behind now ends its queue split and restores the target. Before, the split kept running, with the target patched and its temporary queues on the broker, and the deleted session could stay in `Terminating`. This affects clusters where the operator was upgraded while an SQS or RabbitMQ split session was running. * Ending an Azure Service Bus split in the queue model no longer deletes the scheduled messages in the temporary queue that the deployed workload reads. The split now waits until they are due and the workload has read them, so with `drainTimeout` unset, a message scheduled far ahead keeps the target patched until it is due. Set `drainTimeout` to limit the wait. * In a multi-cluster session, the other workload clusters no longer delete the per-session Azure Service Bus queue or BullMQ queue that the Default cluster created for the session. Before, a session that ended or failed on another workload cluster deleted that queue while the Default cluster still routed messages into it. * Queue splitting now deletes the temporary queues, topics, subscriptions, streams and other broker resources it creates when a session or split ends, also when a session fails to start, after an operator restart, and when the target's namespace is deleted. A BullMQ main queue or GCP Pub/Sub main subscription that still holds messages the workload hasn't read when the split ends is kept instead of deleted with those messages, and is deleted once it is empty. Kafka fallback topics and temporary consumer groups that pods not moved back to the original topics within 10 minutes may still read are deleted once those pods are gone. If the `MirrordPropertyList` or `MirrordKafkaClientConfig` of such a queue, subscription or topic is gone by then, for example with the target's namespace, the operator can't reach it any more, and it must be deleted by hand. * Redis Pub/Sub queue splitting of a sharded channel (`appConfig.shardChannel`) now forwards messages. Before, the operator never finished subscribing to the channel, so neither the deployed workload nor the sessions received any of its messages. * Restart the operator when its session controllers stall. * The operator no longer deletes a GCP Pub/Sub split's temporary subscription, together with the messages the deployed workload hasn't read, when getting an access token takes longer than 2 seconds. It now gets the token before it checks the subscription for messages. It also keeps the subscription when that check fails for a reason other than missing permissions, and deletes it once it is empty. * When a Redis Pub/Sub split ends, the operator now closes its subscriber connections to Redis. Before, it kept one open per channel until the operator restarted. * When a Temporal split fails to start, for example because patching the target fails, the operator now stops polling the real task queue. Before, it kept taking tasks from that queue into a buffer that no worker read, so they waited until Temporal timed them out, and each retry of the session started another poller. * While an ending Azure Service Bus split waits for the deployed workload to read the messages left for it, the operator no longer sends those messages to the dead-letter queue. Each check for messages counted as a delivery, so with the default `MaxDeliveryCount` of 10, Service Bus dead-lettered a message that the workload hadn't read within about 50 seconds, and in the queue model the operator then deleted it with the temporary queue. * With session key header injection on, the deployed Temporal worker no longer receives activity tasks that carry a session's `mirrord-key` header. Before, a task that matched a session but went to the deployed worker, because the session ended or its buffer was full, carried that header. ### 3.233.0 - 2026-11-16 #### Added * The license server serves an OpenAPI description of its report, adoption and deployment endpoints at `/api/v1/openapi.json`. * The operator now logs a panic as an error event, so it appears in JSON and OTLP logs and not only as plain text on stderr. #### Changed * A Kafka queue split using the librdkafka client gives up on a message it cannot deliver after about fifteen minutes (ten minutes of retries after librdkafka's default five-minute `message.timeout.ms`), then ends and hands the target back its own topics. #### Fixed * A CronJob preview that fails because its CronJob name is already taken no longer suspends the CronJob that holds the name. * A Kafka queue split no longer fails every session on it when one session leaves while the brokers are unreachable. * A `mirrord exec` session or preview environment whose queue split cannot be resumed after an operator restart, including when its Kafka branch fails to rejoin, now ends with the split's error. * A `mirrord exec` session, preview environment, or copy target whose queue split fails now hears of it even when the Kubernetes API is briefly unreachable or the operator restarts first. * A copy target whose queue split fails while it runs, or whose split ends because its `MirrordSplitConfig` is deleted, now ends every session running on it with the reason and removes the copy pod, so the workload goes back to normal; a session that connects while the copy pod is going away is refused. * A failed CronJob preview now also stops the jobs it had started and that have not finished, instead of letting them run on. * A preview environment now fails, and a `mirrord exec` session ends with a reason, when its split ends because its `MirrordSplitConfig` is deleted. * A preview whose queue split fails is now marked failed instead of staying ready * A session that fails to join a queue split no longer leaves message filters behind, so the deployed consumer keeps getting the messages they matched and the split's other sessions keep running. * Failed branch messages keep the database container's last output when it exits with code 3. * Fixed Azure Service Bus queue splitting moving messages to the dead-letter queue of a split queue or subscription when the operator's connection to Service Bus kept dropping: the operator now retries a failed forward before it gives the message back to Service Bus. * Fixed an Azure Service Bus queue split stopping, and failing its sessions, when the operator's connection to Service Bus dropped while it forwarded a message. * Kafka splits no longer fail when the brokers become unreachable for a while * MSSQL branches wait up to three minutes for their SQL Server to start and name its log on failure. * Operator-hosted serverless sessions (`operator.sessionsManager: true`) now require the Enterprise plan; with a Team or Free license the Operator refuses them as not licensed and `mirrord operator status` hides the feature. MetalBear Cloud serverless sessions are unaffected. ### 3.232.0 - 2026-11-14 #### Security * Multi-cluster child session connects count only for callers allowed to create cluster sessions. #### Infrastructure * The `mirrord-operator-envoy-remote` role gains `patch` on `mirrordclustersessions`, so the Primary can write a consumer cluster's split details on the Default cluster's child session. * The share-ingress cluster role gets get, list and watch on `mirrordclustersessions` cluster-wide to route share links, so share-ingress can read each session's owner. The mirrord user roles get the `get` verb on `sessions` in the `operator.metalbear.co` API group, which `mirrord exec --share` uses to read its link. #### Added * Added share links for local sessions: with share-ingress set up, `mirrord exec --share` prints an HTTPS link that reaches the developer's local version of the target through the target's Service of the same name. * Kafka `jq_filter` now works with the Java client sidecar. * Multi-cluster queue splitting can be pinned to one cluster with `MirrordSplitConfig.spec.consumerCluster`: only that cluster splits the queues and patches its workload, and the local application still gets the split queue names through the Default cluster. * Queue splitting reads config files mounted from Secret volumes too; a `secret` source needs `valueSelector` or `valuePattern`. * The operator reports the size of its Kubernetes caches, in `mirrord_operator_cache_size_bytes` and `mirrord_operator_cache_objects`, and the bytes of mirrord-protocol messages queued for each user's sessions, in `mirrord_operator_buffered_messages_bytes`. #### Changed * A split session's version of a config file from a `podFile` source is kept in a Secret owned by the session instead of the session's status. * Default branch images moved to supported versions: MySQL 8.4, ClickHouse 26.8, DynamoDB local 3.3.2, Spanner emulator 1.5.59, Flyway 13, Kafka 4.4.0. Before upgrading, an install whose `dbPod.image.registry` or `initImage.registry` points at a mirror needs the new tags mirrored, or `version` pinned on every branch; otherwise branches with no `version`, and pods rebuilt after node loss, fail with `ImagePullBackOff`. A `migrationImages.flyway.registry` mirror always needs the `13` tag, whatever the branches pin, or Flyway `path` migrations without their own `image` fail the same way. A MySQL registry pointed at `mysql/mysql-server` has no `8.4` tag; point it at `mysql` or pin `version: "8.0"`. A source on the old default (MySQL 8.0, ClickHouse 24.8) whose branches set no `version` now gets a newer branch, and mirrord warns about the version mismatch on every session of such a branch; pin `version` to keep the branch matching the source. MySQL 8.4 removed options that MySQL 8.0 took, among them `--default-authentication-plugin` (which the chart example and the 3.197.0 release notes once suggested for `dbServerArgs`), `--skip-host-cache`, `--expire-logs-days` and `--binlog-transaction-dependency-tracking`; an 8.4 branch exits on any of them, so drop from `dbServerArgs` what 8.4 removed, or pin `version: "8.0"` on the branches. The 8.4 replacement for `--default-authentication-plugin=mysql_native_password` is the pair `--loose-mysql-native-password=ON` and `--authentication-policy=*:mysql_native_password`: 8.4 refuses the policy unless the plugin is loaded, and MySQL 8.0 and 9 exit on the policy, so set the pair only where every branch runs 8.4. * Java client Kafka forwarding for plain consumers keeps each record's source timestamp, matching Kafka Streams and the librdkafka client. A broker with `log.message.timestamp.before.max.ms` set rejects an old backlog, which stops the forwarder. * The `mssql-tools` image no longer ships Microsoft's `sqlcmd` and `bcp`, whose license forbids publishing them. MSSQL branches export table data with the `bcp` in the branch's own SQL Server image, in a new `mssql-branch-dump-data` init container, so a custom branch image needs `bcp` to copy table data. The operator pulls `ghcr.io/metalbear-co/mssql-tools:` instead of `:latest`. * `consumerCluster` refuses `secret` volume and `podFile` sources off the Default cluster. The primary operator does the refusing, so upgrade it before the other clusters: an older primary lets such a split start, and the local application then reads the original queue names. #### Fixed * A Kafka `jq_filter` on the Java client that calls a function its jq 1.6 engine lacks now fails the session at start with the function named, instead of matching nothing. * Agent pods left behind by a lost create call are removed after the startup timeout. * Agent startup timeouts now say why the scheduler rejected the agent pod. * DNS lookups, file opens and new outgoing connections made while a session has no live mirrord-agent, for example while a target pod is being replaced, now wait for the next agent instead of failing right away. * Fixed the Java client sidecar crashing on headers without a value. * Kafka mirror mode now works with the Java client sidecar. * MySQL branches load the `mysql_native_password` plugin through the settings file every sourced branch starts with (`loose-mysql-native-password=ON`, which MySQL 8.0 and 9 ignore), so a declared user that authenticates with that plugin on the source (the default on RDS MySQL 8.0) can log in to a MySQL 8.4 branch instead of failing it (`roles: full`) or being left without a login (`roles: empty`). Other copied accounts stay locked as before. A `dbServerArgs` entry such as `--mysql-native-password=OFF` still wins over the file. The file reaches the server on images that read `/etc/mysql/conf.d`, as the official `mysql` image does; a branch on an image that does not (Percona's, Oracle's `community-server`) needs `--loose-mysql-native-password=ON` in `dbServerArgs`. * jq message filters no longer time out while waiting for a thread ### 3.231.0 - 2026-11-11 #### Infrastructure * The `PreviewSession` CRD's `spec.sync` replaces `runCommand`, `workingDir`, `install` and `restartOnChange` with `run` (`command`, `workingDir`, `restart`) and `prepare` (`command`, `workingDir`, `triggeredBy`, `persist`), and `status.sync.install` is now `status.sync.prepare`. A session that an earlier `mirrord preview sync` started loses its run command, and runs its image's own command once the operator restarts, so upgrade the CLI with the operator and run `mirrord preview sync` again. #### Added * A preview with code sync but without `sync.run` in `mirrord-up.yaml` runs its image's own command on the synced files, with the container's own probes, for example nginx serving a site that is built locally. * `POST /splitconfigdbbranches` applies a copy mode the session requests to each requested `dbBranches` entry, replacing only its `copy.mode`, and refuses one missing from the entry's `copy.allowedModes`. Advertised as `DbBranchesSplitConfigCopyMode`. #### Changed * A Kafka queue entry whose topics come from one environment variable value, such as a JSON array read with `valueSelector: ".[]"` or a list split with several `valuePattern`s, now fails instead of splitting only the first topic. Each topic needs its own environment variable, or a JSON object of topics. * A `scaleTriggerOverrides` entry that uses `{{.queue}}` of a Kafka queue entry with several topics is now skipped with a warning, since each of the entry's topics gets its own fallback. * A queue split of two Kafka queue entries that resolve to the same topic, including two topic entries of one `MirrordKafkaTopicsConsumer`, now fails instead of feeding both entries from one fallback topic, which broke after an operator restart. * The operator's hourly telemetry heartbeat now includes its memory and CPU usage and the number of sessions, agents, copy targets, split sessions, split configs, database branches, and preview sessions it is handling. It contains only these numbers, no names, and on Enterprise licenses `operator.disableTelemetries` turns it off as before. #### Fixed * A Kafka queue entry in a `MirrordSplitConfig` now splits every topic its references resolve to, not only the first one. * A Kafka queue entry whose group or application id references resolve to different values now fails to split instead of splitting under the first one. * A queue split whose `MirrordSplitConfig` or target changed its queues, topics, subscriptions or consumer groups while the operator was down is now rebuilt when the operator restarts, ending its sessions so their clients start over, instead of resuming with the old ones. A session asking for a queue added meanwhile rebuilds the split instead of failing. This covers splits started by this operator version or later. * A queue splitting session that fails because its target or split config disappeared, or because the split's queues changed, now has its own temporary queues, topics and subscriptions deleted. Before, they could stay on the broker for good, for example when the target's namespace was deleted while the session ran. * An operator configured with a cloud API key and no license key now syncs the organization's active users from MetalBear, as an operator with a license key does. Seat limits now count every user in the organization rather than only those the operator pod had seen since it started, users who stopped using mirrord free their seats again, and `mirrord operator status` reports the organization's active user counts. * Fixed Azure Service Bus preview recovery after an operator restart when multiple topics use the same subscription name. * Fixed preview queue splits ending when the operator restarts. * Kafka branching now sets a topic's start offset for every group or application id its queue entry's references resolve to, instead of only the first, and fails for an entry with several topics and several ids instead of reading them all under the first id. * Reject single-cluster sessions on a management-only multi-cluster Primary with a clear error. ### 3.230.0 - 2026-11-04 #### Infrastructure * The `PreviewSession` CRD gains `status.sharePort`; apply the updated CRDs with this release for share links to use it. #### Added * Branch config `dbPod.tolerations`, `nodeSelector` and `affinity` for branch pods and Jobs * Kafka branching supports Job and CronJob targets, swapping the copy's bootstrap servers for the branch broker. * `MirrordSplitConfig` queues accept `keyJqFilter` and `enforceFilter`. #### Changed * A `MirrordSplitConfig` naming a Job or CronJob now matches; queue splits on such a target fail with a clear error instead of being silently dropped, and `podFile` refs report that they need a running pod. * Multi-cluster sessions and cross-cluster `mirrord subscribe` streams now require an Enterprise license. A Teams operator with linked clusters configured keeps serving ordinary single-cluster sessions and its own event stream, refuses an explicit request for multi-cluster orchestration, and no longer advertises the multi-cluster features. Copy targets on such an operator are refused as unlicensed, as is a reconnect to a multi-cluster session once the license no longer covers it, and `mirrord subscribe` now needs a valid license like every other operator route. * The operator's Kafka clients no longer send client metrics to brokers that subscribe to them (KIP-714) unless `enable.metrics.push` is set to `true` in the `MirrordKafkaClientConfig`. This also lowers operator memory use per Kafka split. * The share-ingress chart's default `server.logLevel` shows share-ingress's own logs. Set it to include `share_ingress=debug` to also log the upstream each share link request is forwarded to. #### Fixed * A share link whose target can't be reached fails with a 502 after five seconds instead of hanging until the load balancer in front of share-ingress times out. * Cleanup of a temporary broker resource no longer retries every few seconds forever when it cannot succeed (for example when the property list holding the broker credentials was deleted together with its namespace). After an hour in deletion the record reports `Ready: False` with reason `Orphaned` and is retried hourly, so the leak is visible instead of buried in log noise. * Ending a queue split now waits for the target to finish restarting instead of force-deleting its patched pods. * Kafka split sessions wait for the forwarder's partition assignment before reporting ready. * MySQL and MariaDB branch copies no longer fail when another service drops a table mid-dump: the dump runs as a single transaction instead of locking tables, reruns on a vanished table, and `excludeTables` leaves transient tables, and the views reading them, out entirely. Dump args that lock tables (`--lock-all-tables`, `-x`, `--lock-tables`) keep locking, without the single transaction. * Outgoing connections to Unix sockets and localhost now always reach the same pod when targeting a workload with multiple pods. * Queue split message filters written as `.*.*` use less operator memory per session. * Queue splitting across many workloads no longer makes operator memory and Kubernetes API traffic spike when many sessions start or end together. * Queue splitting uses less operator memory and fewer Kubernetes API requests when a namespace holds many `MirrordSplitConfig`s. * RabbitMQ queue splitting now declares typed queue arguments such as `x-single-active-consumer` (boolean) and `x-message-ttl`/`x-max-length` (integer) with the type RabbitMQ requires, instead of as strings that RabbitMQ rejects. Temporary quorum queues are now deleted at teardown instead of being left on the broker. * S3 branches no longer fail with "the operator restarted while the branch bucket was being prepared": a branch whose bucket clone was lost, to an operator restart or to a dropped API reply, is cloned again instead. * Share links reach previews whose target's Service exposes the stolen container port under another port (such as `80` for a container's `8080`). This needs both the operator and the share-ingress chart at this version. ### 3.229.0 - 2026-11-01 #### Infrastructure * For preview code sync, when `operator.previewEnv` is on: * The `mirrord-operator-user` and `mirrord-operator-ci` ClusterRoles, and the namespaced `mirrord-operator-user` Roles, gain `get` and `list` on `previews/sync` in API group `operator.metalbear.co`, the websocket that `mirrord preview sync` pushes code through. * The operator gains `create` and `get` on `persistentvolumeclaims`, `get` on `storageclasses` in API group `storage.k8s.io` and `patch` on `secrets`. It patches only the sidecar TLS Secret that a deleted preview of the same name left behind. * The operator container and the operator Service expose port 4980, which the sync sidecars dial, also outside a multi-cluster setup, and the operator creates the CA Secret of the gateway on that port. * The `PreviewSession` CRD gains `spec.sync` and `status.sync`, and `spec.image` may be empty for a session with `spec.sync`. #### Added * Preview sessions can sync local code: `mirrord preview sync` pushes a service's `sync` directories into its preview pod through the operator, where a sidecar writes them to a code volume that the app container mounts, and a supervisor runs the app and its install command. A sync session without an image of its own, which `mirrord preview sync` creates for a service without `--image`, runs the target's own image. The `operator.preview.codeSync` chart values set where that code lives (a claim of each session, 2Gi by default, or an `emptyDir`), the most bytes that one push may write, the fsGroup of the code volume and the resources of the sidecar. Code sync needs Kubernetes 1.29 or newer, on the API server and on the nodes. #### Fixed * A queue split whose forwarder stops now fails its sessions and restores the target, instead of staying active with nothing forwarding messages. * Fixed a bypass of workload-scoped mirrord policies, including Deployment policies, when targeting Services or using label selectors. * Fixed workload selector expressions being ignored during pod selection, watches, pod replacement, and queue-splitting readiness checks. * Target lookups now reject serverless targets, including ones that name a replica. ### 3.228.1 - 2026-10-29 #### Fixed * A preview session's `Ready` condition now names what keeps its pod from starting (for example an image pull back-off) and carries the failure message once the session fails, instead of staying on "waiting for the preview pod to become ready". * The operator no longer uses gigabytes of memory when a namespace holds hundreds of `MirrordSplitConfig`s: checking the configs for conflicts and tearing down their sessions on deletion read from the operator's cache instead of listing every config or split session once per config. ### 3.228.0 - 2026-10-29 #### Added * Added support for configuring the agent's HTTP/2 header size limit with `agent.extraConfig.http2_max_header_list_size` in bytes. #### Fixed * SQS and S3 requests retry a dropped connection for longer, so a brief endpoint hiccup no longer fails a queue split session at startup. ### 3.227.0 - 2026-10-28 #### Security * Updated the bundled Cloud SQL Auth Proxy used for Cloud SQL database branching to 2.26.3, which fixes known vulnerabilities in its Go runtime and dependencies. No known impact or exploitation in mirrord. #### Infrastructure * Added cluster-scoped `create` permission for `featuregatehits` in API group `operator.metalbear.co` to the `mirrord-operator-user`, `mirrord-operator-ci`, and `mirrord-operator-user-basic` ClusterRoles. Custom RBAC must include this permission to allow CLI paid-feature refusal reports. #### Added * Added reporting for paid-feature refusals on the free plan, including refusals detected locally by the CLI, and telemetry when a Teams license falls back to the free plan. #### Changed * The operator no longer sends `operator.clusterName` to MetalBear when telemetry is anonymized or data sharing is off. It is still sent to a self-hosted license server. #### Fixed * The Java Kafka sidecar no longer runs out of memory when it authenticates to MSK with IAM. Kafka clients share AWS credentials providers instead of leaking one per client, and the sidecar exits on `OutOfMemoryError` so that Kubernetes restarts it. ### 3.226.0 - 2026-10-27 #### Security * Updated dependencies to fix the security advisories reported against them. #### Added * Redis branches with `cluster: true` run as a one node Redis Cluster owning every hash slot, so Redis Cluster clients can use them; `copy.mode: all` copies from a standalone or a Redis Cluster source. * The operator chart accepts `operator.kafkaSplittingSidecar.extraEnv` to set additional environment variables on the Java Kafka splitting sidecar. #### Changed * The cloud API key now takes precedence for licensing, with fallback to other configured license credentials. #### Fixed * Fixed `roles: full` branches replacing an app user that comes from a Secret with `root`. * The operator now reloads its license file when it changes, without a restart. ### 3.225.0 - 2026-10-24 #### Added * Added `dbPod.copyDeadlineSecs` to the branch config: the longest a branch copy Job may run before it is stopped and the branch fails (default one hour). * Preview sessions that mark their queue splits as defaults (`autoQueueSplitting`, sent by `mirrord preview up`) skip the brokers whose splitting is off instead of failing, as `mirrord up` sessions do. #### Fixed * A database branch whose copy step runs longer than its `ttl_secs` is no longer deleted while the copy is still running: the running copy keeps the branch alive, the same way an active session keeps a ready branch alive. A copy that runs past its deadline, one hour by default, is stopped and fails the branch, so a hung copy no longer leaves its branch behind. * A queue splitting session no longer stays in `ResourcesReady` after it is ready when the write of its `Ready` status gets no answer from the Kubernetes API. Each status write now gives up after 15 seconds and is retried, and failed writes are logged at warn level. * A queue splitting session that joins a split started before one of the `MirrordSplitConfig`'s queues was visible to the operator no longer fails with "queue not found in resolved params": the split is rebuilt with every queue of the config, and the sessions already on it reconnect. * DynamoDB branches on clusters with a default storage class no longer fail to start. The branch pod hands its data volume to the database user with `fsGroup`, so the database can open its files on a provisioned volume. * MSSQL branches on clusters with a default storage class no longer fail to start. The branch pod hands its data volume to the database user with `fsGroup`, so SQL Server can copy its system files onto a provisioned volume instead of exiting on "Access is denied". * Starting a mirrord session no longer fails with a 500 "already exists" error when the operator is slow to see the session it just created: a resent start request now waits for the session the first attempt created instead of failing on its name. * The `Drained` condition a split teardown records on a `MirrordSplitConfig` no longer disappears: the operator rewrote the condition list from an outdated copy when it rechecked the config, and could drop a `Drained` written moments earlier. * Waking idle previews tolerates image pull failures until `wake_timeout_secs`. * `tmpNameTemplate` in a `MirrordSplitConfig` names the temporary queues and topics for RabbitMQ, SQS, and Kafka too, and for Kafka takes precedence over `kafkaSplittingTopicFormat`. It supports `{{RANDOM}}`, `{{FALLBACK}}` (`-main-` for the queue the patched workload reads, `-` for a session's) and `{{ORIGINAL}}`; SQS FIFO queues keep `.fifo` at the end. With `mirrord.temporary_group_id`, Kafka sessions' consumer groups follow the template too, with `{{RANDOM}}` derived from the session id; sessions running during the upgrade switch to their templated group and start from its reset offset. A config with RabbitMQ, SQS, or Kafka queues is no longer accepted when its template is missing `{{RANDOM}}`, uses another placeholder (including Kafka's `{{ORIGINAL_TOPIC}}`), contains characters the broker does not allow in names, or is too long for the broker's name limit. Permissions or ACLs scoped to the `mirrord-` prefix must also cover the names the template produces. ### 3.224.0 - 2026-10-22 #### Security * Update `webpki-roots`, `webpki-root-certs` and `rustls-webpki` to stop trusting the compromised eMudhra intermediate certificate authorities. #### Added * DB branches read connection values from AWS SSM Parameter Store. * Teams operators allow `mirrord ci` without a CI API key while retaining licensed Teams features. Free and Teams single-cluster CI attach to at most one ready target pod per connection. * The operator dashboard shows a prompt to paste into an AI coding agent, which then installs mirrord and sets it up. #### Changed * Advertise features that the Helm values switch off, so the CLI can tell users to enable the value instead of upgrading the operator. * PostgreSQL branch errors explain unreachable sources and unverifiable `sslmode`. * The dashboard login and sign-up pages have their own browser tab titles. The default page title is "mirrord for Teams Dashboard | MetalBear". #### Fixed * Added `containerNames` to `MirrordSplitConfig` app-config references to avoid Kustomize `images:` build failures. The legacy `containers` field remains supported; use only one spelling per reference. Upgrade both the operator and CRD before adopting `containerNames`, and restore `containers` before rolling back to an older operator. * Fixed SQS queue splitting filters not matching SNS `Number` message attributes that carry a custom data type label, such as `Number.java.lang.Integer`. * The operator status reports the header filter each preview environment was started with, so the browser extension joins a preview that filters on a custom header with that header instead of `baggage`. ### 3.223.0 - 2026-10-19 #### Infrastructure * Opt-in `dbBranchNamespaces` grants `get`/`list`/`create`/`patch`/`delete` on namespaces for cluster-wide installations, `list`/`patch`/`delete` on ConfigMaps, and `get`/`create`/`patch` on ServiceAccounts #### Added * Run DB branches in a dedicated namespace per session key, with a configurable `operator.dbBranchNamespacePattern` (default `mirrord-{{key}}`); existing branches retain their namespace when the pattern changes ### 3.222.0 - 2026-10-18 #### Infrastructure * A `BranchDatabase` spec can no longer change after the branch is created, except for `migrations`. * With `operator.kafkaBranching`, the chart adds the `MirrordClusterKafkaBranch` CRD and lets the operator manage Services in its own namespace. * With `operator.sessionsManager` enabled, the `mirrord-operator-user`, `mirrord-operator-ci` and `mirrord-operator-user-basic` ClusterRoles are granted `proxy` on `serverlessclientassignments` and `get` on `serverlessdataplanes` (`operator.metalbear.co`), the resources a developer's mirrord uses on the operator-hosted sessions-manager, and a new `mirrord-operator-sessions-manager-agent` ClusterRole is created for the workload companion's agent with `proxy` on `serverlessagentassignments` and `get` on `serverlessdataplanes`. Setting `operator.sessionsManagerUserRbac` to `false` leaves the client rules out of the mirrord user ClusterRoles, for granting scoped client access through your own roles. #### Added * Added Kafka branching, which runs a session's Kafka queues against an isolated broker shared by its key and wakes idle preview environments. * DB branching selects one field from JSON AWS/GCP secrets. * mirrord policies can block database branching, for all branches or by engine and source name, and restrict branch copy modes. #### Changed * Dashboard: the topology "Data store" category is now called "Database". * Dashboard: the topology map opens with Entry point and Preview env categories hidden; the legend chips turn them back on. * Dashboard: the topology map routes edges between namespaces for up to six open namespaces instead of three, so cross-namespace connections stay orthogonal on mid-sized clusters. * Dashboard: the topology toolbar wraps instead of cutting off legend chips, namespace, highlight and preview env key filters live under an Advanced filters toggle, and an empty time range says so instead of claiming topology is off. #### Fixed * An idle preview environment whose target has no queue to split, and no incoming ports, fails with a missing wake source instead of sleeping until its TTL. * Dashboard: topology namespace labels no longer spill past their lane when the map is zoomed out. * Preview environments and copy targets set split queue env vars in every container the `MirrordSplitConfig` names, not just the target. ### 3.221.0 - 2026-10-14 #### Added * A `MirrordSplitConfig` can point KEDA triggers that measure a queue through a query at the split's temporary queue with `scaleTriggerOverrides`. #### Changed * Temporal task queues always split in `steal` mode, ignoring `mirror` requests. #### Fixed * Preview sessions that lose their target pods now report the no-live-agent timeout and how to raise it, instead of "agent disconnected unexpectedly". * Replacing a preview with a new image keeps its share host and link. * Sessions no longer time out while a replacement target pod's agent is still starting. ### 3.220.0 - 2026-10-13 #### Infrastructure * The operator role gains `create` and `delete` on Services for the per-branch Service, and `patch` on PersistentVolumeClaims to mark a completed first load. * `mirrord-operator-envoy-remote` can watch `events` so the primary relays member clusters' events. * `mirrord-operator-user-basic` no longer grants `create` on `previewsecretmounts` and `splitconfigdbbranches`; the `roleNamespaces` Roles do. #### Added * A `MirrordSplitConfig` can select workloads by label with `targetSelector`, and every config that applies to a workload is merged. * MySQL and MariaDB branches can recreate source accounts with their grants and definers via the `dbPod.roles: full` chart setting. * Supporting Free-tier operators allow `mirrord ci` without a CI API key, using automatic ordinary client credentials and the same single-pod-per-connection limits, feature restrictions, and compatible concurrent-session behavior as Free `mirrord exec`. * `mirrord subscribe` on a multi-cluster primary streams every linked cluster's events. #### Fixed * Database branch pods deleted or evicted are replaced on their existing volumes, behind a Service that keeps sessions connected. * MySQL 5.7 branches leave out stored routines the source user cannot read and warn about them, instead of failing. * Preview `secret_mounts` Secrets are created only for an existing `PreviewSession`, and never replace a Secret holding other contents. * Return Kubernetes type and object metadata from the database branch credentials API while retaining the response fields used by older clients. * Users bound to `mirrord-operator-user` by a `RoleBinding` can use literal DB branch connection values. ### 3.219.0 - 2026-10-11 #### Infrastructure * Added `identities` to `agentAsClient` section in the `MirrordTlsStealConfig` and `MirrordClusterTlsStealConfig`, letting the agent present a different client certificate to the target depending on the identity of the original client. * Branch database CRDs gain `status.sessionInfo.*.instanceId`; apply the updated CRDs with this release. * Keep webhook list and delete for older operator images. * The chart installs the `mirrord-operator-pod-mutator` MutatingWebhookConfiguration and grants `get` and `patch` on that one name. `list` and `delete` stay so an older operator image can finish startup cleanup. * The license server chart now applies `database.port`, and sets a Postgres SSL mode the server accepts: `require` when `database.tls` or `database.auth.awsIam` is set, instead of a misspelled `verify-full` that was ignored and fell back to `prefer`. New `database.sslMode` and `database.sslRootCert` values enable certificate verification, e.g. `verify-full` with the RDS CA bundle. #### Added * Database branch entries on a MirrordSplitConfig resolve `db_branches: "*"` / ids and are shared per session key. #### Changed * Migration and copy Job pods inherit `dbPod.labels` and `dbPod.annotations`. * Operator registers mutating webhooks only for queue splitting. #### Fixed * A database branch whose expiry passes while a session still uses it gets its TTL extended, instead of the operator re-checking it continuously. * Database branches expire by their TTL after an operator restart instead of being kept alive by the sessions that ended with it. * Database branches whose pod or migration Job pod the cluster rejects expire by their TTL, releasing their PersistentVolumeClaims instead of keeping them indefinitely. * In multi-cluster setups, branch statuses on the primary cluster drop ended sessions and cleared errors, so `mirrord db-branches status` stays current. * MySQL and MariaDB `dump_args` can leave out objects a copy mode includes, such as stored routines with `--skip-routines`. * MySQL and MariaDB branches leave out stored routines the source user cannot read and warn about them, instead of failing. * MySQL branches load in every copy mode from sources with GTIDs enabled. * Queue splitting reports why target pods aren't ready while a session waits on them, and fails the session at once if they can't start. * When no pod of a session's target is ready, the session error names the pods and why, instead of only saying the target could not be processed. ### 3.218.0 - 2026-10-07 #### Infrastructure * The MirrordSplitConfig CRD schema accepts `dbBranches`, and the CRD plus the operator's `mirrordsplitconfigs` permissions are installed when only database branching is enabled. * The `PreviewSession` CRD accepts `spec.specResources`: a pod template, ConfigMaps, and Secret key references. #### Added * Added `dbBranches` to `MirrordSplitConfig` for declaring database branches per workload * Previews can be built from manifest files with `mirrord preview start --resource`. #### Fixed * Clients reconnect as soon as a split's queues change. * PostgreSQL branches keep the PGDATA a custom image sets. * Queue splitting now restores the target's pods before deleting the shared temporary queue they read from, so ending a session no longer crashes pods that treat a missing queue as fatal. A GCP Pub/Sub main output subscription that still holds messages at that point is kept, and named in the `Drained` condition, instead of being deleted with them. Failed or crash-looping patched pods are replaced right away instead of after a timeout. * The "Read the docs" link on the empty Topology tab opens the Topology docs page. ### 3.217.0 - 2026-10-06 #### Infrastructure * The `PreviewSession` CRD gains an optional `spec.dbBranching.connectionSources` field, the session's own env var mapping for a reused db branch whose spec was written by a different config. * The rules in the `mirrord-operator` ClusterRole are in a different order, and some rules are split in two. They give the same permissions as before. * With `namespaced: true`, the `mirrord-operator` ClusterRole keeps only the rules for cluster-scoped resources. The chart adds a `mirrord-operator-namespaced` Role and RoleBinding with the rules for namespaced resources in each namespace of `allowedNamespaces` and in the operator namespace. With the default `namespaced: false`, the permissions do not change. #### Added * New `namespaced` and `allowedNamespaces` chart values limit the operator to some namespaces. The operator refuses sessions in other namespaces. * Share-ingress chart option `shareIngress.notFound.redirect` (default `true`). Set it to `false` to serve a plain "Preview not found" 404 page with no redirect, refresh, or JavaScript for share hosts without a live preview; `shareIngress.appDomain` is then optional. #### Fixed * Fixed queue splits stopping for live previews when the API server is overloaded. * Preview pods reusing a db branch get their own env var mapping, with branch values delivered via Secret. ### 3.216.0 - 2026-10-05 #### Infrastructure * Add `spec.incoming.tlsDelivery.clientAuthFromTarget` to the `PreviewSession` CRD. When it is set, or when the target's `MirrordTlsStealConfig` sets `agentAsClient.authentication`, the operator reads the TLS client certificate and key from a running target pod through an agent and keeps them in memory for the session only. #### Added * Preview sessions can present a TLS client certificate that already lives in the target's container (`tls_delivery.client_cert_source: target`): the operator reads it from a running pod of the target instead of requiring a copy on the developer's machine. Without any client certificate in `tls_delivery`, previews now present the agent's client certificate from the target's `MirrordTlsStealConfig` (`agentAsClient.authentication`). A certificate read from the target is read again every hour, so previews keep working after a sidecar renews it in place. #### Fixed * Database branches now start on PostgreSQL 18 images. ### 3.215.0 - 2026-10-05 #### Security * Hide the values of headers and message properties with names that often hold secrets, such as `Authorization`, `Cookie`, `X-Api-Key` and `Client-Secret`, in the operator's request and message logs. The logs show `[REDACTED]` in place of these values. #### Infrastructure * Allow restricted roles to read and connect to generated copy names. The operator checks the caller's permissions for the original target. * Allow the operator chart to create namespaced roles and bindings for access to specific targets. For operations on copy pods, access to the original target is used. * ScaledObject get, list, watch, and patch are granted by `operator.manageKedaScaledObjects`. * The `mirrord-hooks` ClusterRole of the `pre-delete` hook is granted list, get, watch, delete and patch on `mirrordclusterworkloadpatchrequests.mirrord.metalbear.co`, so that the hook can clear workload patch requests during `helm uninstall`. #### Added * Organization admins can edit and reset user display names in the cloud dashboard, including labels for anonymized users. Manually assigned names are preserved when new session telemetry arrives. * Set the `operator.hideHeadersAndProperties` Helm value to `true` to remove HTTP request headers and queue message properties from the operator's `Message Processing` logs. #### Changed * Split targets scaled by KEDA scale on the temporary queue they consume, down to zero, instead of staying at one replica. #### Fixed * A Flux Kustomization or HelmRelease is suspended for a rescaled workload only if it applies that workload, not merely because the labels name it. * A copy target scaling down a KEDA-scaled workload keeps it at zero instead of KEDA scaling it back up, when `manageKedaScaledObjects` is set. * Argo CD applications paused for a scaled-down copy target resume even when the operator restarts during the session. * Database branch pods pull private images with the target's `imagePullSecrets`, so a custom branch `image` needs no extra configuration. * Fixed cleanup of queue resources for sessions started after an operator restart. * On `s3_event` SQS queues, only jq filters naming `S3Metadata` fetch and see it, so other sessions keep receiving messages when S3 cannot answer. * `helm uninstall` no longer leaves the `mirrordclusterworkloadpatchrequests.mirrord.metalbear.co` CRD stuck in deletion when a mirrord session was still ending, and no longer leaves a workload patched when its patch was removed before the session that patched it. ### 3.214.0 - 2026-09-30 #### Infrastructure * The BranchDatabase CRD schema accepts `additionalDatabases` in `postgresOptions`. * `operator.previewEnv` defaults to `true`, so a default installation includes the `previewsessions.preview.mirrord.metalbear.co` CRD, and the operator's own `mirrord-operator` ClusterRole (not the user role) gains the preview environment permissions: creating and deleting Services, Deployments, CronJobs and Jobs, managing PreviewSessions, and creating ConfigMaps and Secrets for preview sessions. In multi-cluster setups, the workload-cluster role also gains `get`/`create` on Secrets. Set `operator.previewEnv: false` to keep the previous CRDs and roles. #### Added * GCP Pub/Sub queue splitting: set `gcs_event: "true"` in the queue config so jq filters can match the custom metadata of the Cloud Storage object a notification is about, as `.gcsMetadata`. Notifications with the `NONE` payload format also need `storage.objects.get` on the bucket for the identity of the queue's client config (its `credentials_json` service account, or the operator's own identity). * PostgreSQL branches copy several databases from one source server into one branch pod. #### Fixed * Karpenter can now replace or remove a node that runs a targeted mirrord agent, for example in a Kubernetes upgrade. The agent pod has its node as the owner, and it stops together with the node. `kubectl drain` also evicts targeted agent pods without `--force`. ### 3.213.0 - 2026-09-30 #### Infrastructure * Enabling `operator.turbopufferBranching` grants the operator the database branching Secret permissions and installs the `BranchDatabase` CRD, as the other branching dialects already do. * The PreviewSession CRD schema accepts a label selector target. #### Added * Added turbopuffer namespace branching. * Preview sessions accept a label target covering several workloads. #### Fixed * BullMQ queue splitting no longer loses a job when the operator restarts or Redis fails while the job is being forwarded. * Preview `config_mounts` and `secret_mounts` now work at paths inside a directory the target mounts from a ConfigMap, Secret, or projected volume. * RabbitMQ queue splitting no longer deletes a temporary queue that still holds unconsumed messages or has a consumer attached, so deliveries a pod holds unacked are not lost with it; the split waits for `drainTimeout` and reports the kept queue in a `Drained` condition. * RabbitMQ queue splitting no longer loses messages when a session or main output queue is deleted mid-session; they stay on the original queue until the queue is back. * The Kafka splitting Java sidecar can now consume from topics compressed with zstd, snappy, or lz4. The sidecar gets a writable `/tmp` to unpack the codecs' native libraries into. * The Kafka splitting sidecar no longer leaks admin clients. ### 3.212.0 - 2026-09-25 #### Infrastructure * The operator ClusterRole gains `get` on Secrets to read a preview session's TLS client certificate from the Secret it created. The multi-cluster Envoy role also gains `get` and `create` on Secrets when preview environments are enabled, to replicate session-owned credentials to workload clusters. #### Added * Database branch connection params layer over a `url` param, so one connection string can supply the host, port and database. * Failed database branches now report the branch pod's last log lines in their error and Ready condition, so users see why branching failed without reading pod logs. * Preview sessions present the `tls_delivery.client_cert` client certificate when delivering stolen TLS requests to preview pods that require one. Startup waits for the CLI credential upload, and session Secrets follow previews to the default cluster and additional replicas. The operator advertises preview TLS delivery support so newer CLIs can reject incompatible installations before replacing a session. * Queue splitting sessions accept composable message filters (`metadata` regexes combined with `any_of` / `all_of`) next to the existing `message_filter` map. * The mirrord UI now shows how many incoming traffic deliveries each active port subscription receives. #### Changed * The operator chart's missing-credential error points AI coding agents at the agent-started trial. #### Fixed * License errors now point at the MetalBear community Slack and the `hi@metalbear.com` address, replacing a link to the retired Discord server and an outdated email domain, and break across lines instead of printing a literal `\n` between the message, the reason, and the contact details. * Source database URLs with a `jdbc:` prefix now parse instead of failing with a missing host. ### 3.211.0 - 2026-09-23 #### Infrastructure * The operator ClusterRole no longer grants `get`/`create` on `pods/exec`. * The operator's ClusterRole is granted `patch` on `mirrordkafkaephemeraltopics`, which it needs to clear the finalizer on leftover objects from an older operator version. Only rendered when Kafka splitting is enabled. * When `operator.topology` is enabled, the operator's own `mirrord-operator` ClusterRole (not the user role) gains `get`/`list`/`watch` on `discovery.k8s.io` `EndpointSlices`, used to resolve which Service a session's outgoing connection reached. The value is off by default, and the role is unchanged while it is off. * With the new `operator.topology` chart value enabled (off by default), the operator records service-to-service edges from the outgoing connections a session's target opens and the incoming connections it accepts, observed during mirrord sessions and preview environments, and serves the resulting topology through the new read-only `mirrordclusterservicegraphs` aggregated API resource (in-memory only, nothing persisted to etcd, no CustomResourceDefinition installed). While enabled, the `mirrord-operator-user` ClusterRole gains `get`/`list` on `mirrordclusterservicegraphs`. #### Added * Added a Topology tab to the dashboard: a service map of the connections observed during mirrord sessions, with the sessions and users behind each edge and node categories derived from well-known ports. Edges are routed around services and bundled, large clusters open folded into one box per namespace, and the map offers search, namespace and category filters, a focus mode for a service's upstream and downstream, a sortable list of every connection, and PNG export. * The license server stores the in-cluster services each session's target, and each preview environment's target, exchanged connections with, and serves them aggregated with a direction as `GET /api/v1/reports/topology`, the data behind the dashboard's topology view. * The operator now reports on its startup telemetry whether it authenticated with a cloud API key, a license file, or a license key. * When `operator.topology` is enabled, session and preview environment telemetry reports carry the in-cluster services the target connected to and was connected from (`connected_services`, each with a direction), so usage dashboards can show a service topology. A connected service that fronts a preview environment also names the preview's key and target (`preview`), so the topology can draw preview environments as such. The field is identity data: the license server receives it as-is, and the cloud receives it only when identity sharing is enabled for the operator's API key. * mirrord now warns when a database branch's image runs a different server version than the source database, naming both versions. #### Changed * Queue splitting reads `podFile` sources through a mirrord agent instead of `pods/exec`, so the container needs no shell or `cat` binary. * The dashboard fills the full viewport width. The embed CSS template's white-label hook is now `.dashboard-embed::before` instead of `.max-w-7xl::before`; embedded dashboards using a copied template need the new selector. * The operator dashboard header shows only the mirrord brand; the "Utilization" subtitle is gone. #### Fixed * A workload patch waiting for a pod that has not rolled away is now polled and logged less often the longer it waits. * Fixed CronJob previews failing for source CronJobs with long names. * Fixed UDP traffic addressed to a target node's stale HostIP being rejected after the target scaled down by rewriting it to a live target node. * Fixed multi-cluster CI sessions not counting towards the concurrent CI session limit. A session started with `mirrord ci start` against a multi-cluster setup is now reported once, from the cluster coordinating it, however many clusters it reaches. * Fixed the cleanup of leftover `MirrordKafkaEphemeralTopic` objects from older operators retrying the same object every 30 seconds for the lifetime of the operator when a step failed. * The operator and license server Deployments no longer render an empty `annotations` field when no pod annotations are set. ### 3.210.0 - 2026-09-21 #### Added * The operator chart accepts `operator.communicationTimeoutMillis`, setting how long either side of a connected session may go silent. * `mirrord subscribe` can now stream every session at once, say how each message was routed, pair an HTTP request with the response that answered it, and report queue messages no session filter took. #### Fixed * HTTPS stealing works when the original client sends no SNI: `agentAsClient.verification.serverName` names the host the agent verifies. * MySQL and MariaDB branches now include the source's stored routines, and copied views, routines and triggers work on the branch. * MySQL and MariaDB branches run with the source server's SQL mode, character set, collation, time zone and transaction isolation. * Queue splitting session startup now reports 500 when the operator cannot watch the split session resources, instead of a misleading 403. ### 3.209.0 - 2026-09-17 #### Added * Preview share hosts can be minted from the session key alone with `operator.shareIngress.stableSlugs`. * The `mirrord_stolen_connections_count` and `mirrord_stolen_requests_count` metrics now carry the `client_user` and `client_name` labels alongside the existing `user_id`, matching the session metrics, so stolen traffic can be attributed without resolving the opaque id. * The usage dashboard has a Preview Environments card and tab: sessions, session time, average duration and peak concurrency for the selected period, daily concurrency against the licensed machine-session limit, usage per service, and every environment with the services it ran. CI sessions get their own view beside Users and Services. The Usage tab is now called General. * The usage report separates preview environment sessions from CI sessions, with session counts, session time, average duration and peak concurrency for each, the combined peak against the licensed machine-session limit and the instant each peak was reached, preview usage per service, the sessions in the reporting period, and daily peaks in the trends. The operator records the preview environment key with each preview session so an environment's services can be reported together. Requires the PostgreSQL backend. * When the organization has agreed to share identities with MetalBear, preview environment sessions reach the cloud usage dashboard with their service, namespace and environment key, including sessions that are still running. Without that agreement they stay anonymized, as before. #### Changed * The preview environments table leaves out sessions recorded without a service or environment and says how many it left out, and the concurrency card shrinks to the billed peak when there is no daily history to plot. #### Fixed * Deleting a Kubernetes namespace no longer hangs when the operator's license has expired. Listing and killing sessions now works whatever the state of the license; starting one still requires a valid license. * Fixed primary buttons in the cloud dashboard, such as Apply on the custom date range, rendering without a background and unreadable in light mode. * The operator reports itself to mirrord cloud as soon as it starts instead of waiting for the first telemetry flush. ### 3.208.0 - 2026-09-15 #### Infrastructure * The `PreviewSession` CRD gains an optional `spec.cronjob.schedule` field, and the operator role (with `operator.previewEnv`) gains `create`, `delete`, and `patch` on `batch/cronjobs` plus `create` on `batch/jobs` to run CronJob previews. #### Added * Added S3 bucket branching (`operator.s3Branching`): a branch of `type: s3` clones the target's bucket in the customer's AWS account - configuration, tags and (optionally filtered) objects - and points the session at the clone. The operator uses its own AWS credentials, so its IAM role needs the S3 permissions listed in the docs. * Preview environments can target a CronJob: the preview is an isolated CronJob running the user's image, triggered once on start and then on its (optionally overridden) schedule. #### Changed * Operator logs now carry the full error chain, so a failure names its cause rather than only the step that failed. #### Fixed * Fixed long namespace names in the dashboard's session activity and service tables wrapping onto several lines or spilling into the next column. * The operator is granted permission to write `MirrordKafkaClientConfig` statuses. ### 3.207.0 - 2026-09-14 #### Added * Added Liquibase migration support. * Redis branch data copy supports `dbPod.sourceSslmode: verify-ca` to verify a TLS source's certificate chain without the hostname check #### Fixed * Fixed queue modes when applied to queues using a wildcard ("\*"). ### 3.206.0 - 2026-09-10 #### Added * NATS pub/sub splitting now honors `tmpNameTemplate` for temporary subject prefixes. #### Fixed * Client certificate signing now works when the cluster is reached through a proxy that validates requests against the cluster's API discovery (such as Teleport). The operator now advertises its client-credential resource in aggregated API discovery, so such proxies recognize the resource kind instead of rejecting the request. * Database branches restore sequence and identity positions, so inserts into a copied table no longer fail on a duplicate id. ### 3.205.0 - 2026-09-09 #### Security * Source TLS material is now mounted only into the copy container, never the branch database container #### Infrastructure * The operator chart grants the `previews/logs` route to users, and the multi-cluster role the `pods/log` access a primary needs to tail each workload cluster's preview pods. * When any queue splitting flavor is enabled, the operator role additionally gets `get`/`create` on `pods/exec` (reading `podFile` sources from running target pods; the websocket exec handshake authorizes as `get`) and `create`/`update`/`delete` on `secrets` (operator-owned Secret copies shadow-mounted over `podFile` sources; user Secrets are never modified). #### Added * DB branching connection params can be read from a ConfigMap entry, including a field inside a mounted JSON or YAML config file. * Queue splitting now resolves queue names from files injected into pods (e.g. by vault-agent-injector) via the new `podFile` source in `MirrordSplitConfig`. * The operator captures the last output from a preview environment's pods when it fails, and serves it to the CLI from every workload cluster. #### Fixed * Fixed Redis branch copy failing with permission denied reading source TLS files * Mounted config file rewrites keep the file byte-identical outside the swapped queue names * NATS split teardown no longer trusts a single momentarily-stale empty-backlog read * Namespaced configuration resources have their status written in their own namespace. * Temporal queue splitting accepts gzip and deflate compressed gRPC requests from workers. * Temporal queue splitting forwards gRPC error details so SDKs raise typed errors. * The port lock conflict error now points at `mirrord operator status` to list active sessions, replacing a suggestion to run `mirrord sessions`, which is not a real subcommand. ### 3.204.0 - 2026-09-09 #### Added * Added core NATS (non-JetStream) pub/sub queue splitting behind the `operator.natsPubsubSplitting` chart flag. The operator republishes subject traffic under per-session temporary subject prefixes with header and jq filters, steal and mirror modes; delivery is best-effort since core NATS persists nothing. * Added the `mirrord.group_join_timeout` Kafka property so `mirrord.temporary_group_id` splits can wait longer than 3 minutes for slow rollouts * The queue-splitting status API now reports the temporary queues created for a split, both the ones shared by the workload and the per-session ones, so `mirrord queues status` can show which broker resources belong to which session. #### Changed * Queue splitting tracks the temporary queues and topics it creates in batches. #### Fixed * Added cluster level route for `Target` and `CopyTarget` so listing resources with `--all-namespaces` option works properly. * Fixed CI session requests failing when a pooled database connection goes stale. * Temporary broker queues and topics are recorded for cleanup before they are created, so a crash cannot leave one untracked. ### 3.203.0 - 2026-09-03 #### Added * Added queue splitting support for NATS JetStream. * Preview environments with database branching now support targets scaled to zero pods. ### 3.202.0 - 2026-09-02 #### Added * PostgreSQL branch pods now honour the `dbServerArgs` branch config. * Sessions from callers that authenticate with a Kubernetes ServiceAccount, such as cloud agents, are now treated as machine sessions: they don't consume a license seat and count against the license's concurrent machine session limit instead. Organizations whose developers authenticate through a shared ServiceAccount can have this turned off in their license, so those callers each take a seat. #### Fixed * Fixed Azure Service Bus per-session subscriptions expiring during long-idle sessions * Preview config mounts now carry session queue names for split files * Preview environments now work on service-mesh-injected targets: the preview pod template is annotated with `traffic.sidecar.istio.io/excludeInboundPorts` (Istio) and `config.linkerd.io/skip-inbound-ports` (Linkerd) for the session's subscribed ports, so the operator's tunnel connections reach the preview app instead of being captured (and, under STRICT mTLS, reset) by the injected sidecar. The sidecar itself is kept, so the preview app's outgoing traffic still flows through the mesh. * Silenced spurious role errors in Postgres branch pod logs. ### 3.201.0 - 2026-09-01 #### Added * RabbitMQ queue splitting now supports jq filters, which run against a JSON representation of the message (`headers`, `properties`, `payload`). This also lets `mirrord up` split RabbitMQ queues automatically, with no queue splitting configuration from the user. * Redis branch data copy now supports TLS source databases #### Fixed * Branch setup container failures now surface on the branch status instead of hanging * Fixed CockroachDB branching failing on source databases containing materialized views, and made copy modes refresh them after the data loads so they are populated instead of empty. * Misplaced database branch config keys now fail loudly instead of being silently ignored ### 3.200.0 - 2026-08-31 #### Changed * mirrord policies are now inherited from resources owning the session target. For example, a policy targeting a deployment also applies to sessions that atrget any of its owned pods. Additionally, specifying a container in the policy target must now be done explicity by including `/container/` in the path. #### Fixed * Fixed AWS Secrets Manager branch sources for targets with static AWS keys. * IAM-authenticated apps connect to Postgres branches without a password. * Postgres and CockroachDB branch copies accept sslmode values in any spelling. * Preview sessions with an HTTP filter now fail when no incoming ports are found instead of silently starting without traffic. ### 3.199.0 - 2026-08-27 #### Added * Added a mirrord policy for enforcing queue splitting filters. * New `operator.pauseKedaScaleIn` holds a queue-split target up and pauses KEDA scale-in for the split, so fallback queues still drain. #### Fixed * Fixed operators configured with a license server trying to authenticate against it with a cloud API key, which failed with a 404 on `/api/v2/token`. Such an operator no longer authenticates to the cloud at all and sends no identities to MetalBear; product telemetry, when enabled, is still sent anonymized. * Fixed the usage dashboard clipping the largest bubble in the session scatter charts and the first letters of long namespace names in the sessions-by-namespace chart. ### 3.198.0 - 2026-08-25 #### Added * Added AWS IAM (MONGODB-AWS) authentication for MongoDB DB branching. * Added AWS Secrets Manager connection sources for DB branching. * Added `operator.annotations` and `server.annotations` to the mirrord-operator and mirrord-license-server charts, which stamp annotations onto the resources the charts create (Deployment, Service, RBAC, and so on) rather than only onto the pod template like `podAnnotations` does. CustomResourceDefinitions are deliberately left out so an ordering annotation cannot delay them past the workloads that need them. This lets GitOps tools order the two charts against each other, for example with `argocd.argoproj.io/sync-wave`. * Copy targets report when they become eligible for deletion. * External resources report when their cleanup keeps failing, and workload patches report whether they are in effect and what they conflict with. * Generic db branches can run a copy Job before turning Ready, with branch defaults and copy configurable via profiles. * Kafka client configurations and queue registries report whether the operator can resolve them. * Legacy Kafka topic consumer configurations now report when a MirrordSplitConfig has taken over their workload. * Policies and property lists report what the operator cannot enforce or resolve. * Resolve queue names from ConfigMaps mounted as files in queue splitting. * Sessions report when they will be closed for being unused. * Sessions, multi-cluster sessions, branch databases and preview sessions report a `Ready` condition. * Split sessions report their lifecycle phase, which operator instance serves them, and when no patched target pod ever became ready. * TLS steal configurations report ports declared more than once, and ports another configuration with the same targets already claims. * `MirrordProfile` and `MirrordClusterProfile` report whether the CLI can apply them, so a bad field reaches the profile's author directly. * `kubectl get` shows phase, target and age for every mirrord resource instead of bare names. #### Changed * Diagnostic print columns such as failure detail, hostname and API version moved behind `kubectl get -o wide`. * Preview share links now work with any HTTP filter, not only the default one. * The operator Helm chart now defaults `operator.jsonLog` to `true`, so operator pods emit structured JSON logs out of the box instead of requiring an explicit values override. #### Fixed * A branch database that recovers no longer keeps reporting the error it hit before becoming ready. * A configuration's `Accepted` verdict now updates when an overlapping or superseding resource changes, and a cluster-wide TLS steal configuration reports conflicts with namespaced ones instead of only with other cluster-wide ones. * A multi-cluster session waits for each cluster to report its own session ready, and reports how long it has been running. * Fixed CockroachDB branch dumps failing on non-public schemas and enum columns. * Fixed MySQL database branching against a source that uses IAM authentication, which failed with "did not become ready" because the MySQL client tools were not allowed to send the IAM token. * Fixed PostgreSQL branch copies failing on foreign keys between copied tables * Fixed Postgres branch init failing when the branch database already exists in the base image. * Fixed Postgres branches reporting ready before the database accepts connections * Fixed Temporal splitting rejecting workflows that schedule activities on the default task queue. * Fixed machine-session duration averages in the license server's Postgres usage report: `EXTRACT(SECONDS ...)` returned only the seconds component of each session's duration, so any session crossing a minute averaged wrong in the report and per-pipeline metrics. * Fixed preview secret mounts when database branching is disabled * Multi-cluster sessions no longer lose their copy target configuration. * Print columns show the target workload, authentication kind and expiry time instead of a serialized object or ``. ### 3.197.0 - 2026-08-24 #### Added * Added operator free tier that has OSS-equivalent feature set, but includes QoL improvements that come with operator (e.g. automatic reconnects, no RBAC requirement for users, etc.) * Operator chart: allow unsetting the default CPU/memory requests and limits by setting them to `null`, and configure agent pod resources first-class via `agent.resources` instead of `agent.extraConfig`. * Postgres branch connections now default `sslmode` to the branch pod's TLS state, overridable per key by admin `dbPod.queryParams` (incl. profiles) and session `query_params`. * Support splitting zstd-compressed Kafka topics. #### Fixed * Fixed private migration images by inheriting the target's imagePullSecrets. * `dbServerArgs` in `mysqlBranchConfig` / `mariadbBranchConfig` (including profiles) is now passed to the branch pod's database server; previously it was silently ignored for those engines and only worked for Redis. Use it e.g. to serve MySQL clients that cannot do `caching_sha2_password` over plain TCP: on 8.0 set `["--default-authentication-plugin=mysql_native_password"]`, and on 8.4 `["--mysql-native-password=ON", "--authentication-policy=mysql_native_password"]`, so the branch's accounts are created with the legacy plugin. Enabling the plugin on its own leaves every account on the plugin it was created with. ### 3.196.0 - 2026-08-19 #### Security * Updated the `h2` dependency to 0.4.16, which fixes RUSTSEC-2026-0258: earlier versions accept and queue empty HTTP/2 DATA frames without limit, so a stream that is not actively drained can grow memory unboundedly or panic on length overflow. #### Fixed * Fixed Kafka queue splitting with `mirrord.temporary_group_id` rejecting workloads that consume several topics with one consumer group. * Granted mirrord users permission to create preview secret mounts ### 3.195.0 - 2026-08-18 #### Security * Released images are now published with signed SLSA build provenance, so you can verify that the image you pulled was built by our release pipeline from this repository. Verify with `gh attestation verify oci://ghcr.io/metalbear-co/operator: --owner metalbear-co`. #### Added * Added mirrord.temporary\_group\_id Kafka splitting property supporting KafkaJS consumers * Preview environments that only use queue splitting can now target workloads scaled to zero replicas, e.g. consumers scaled down by an autoscaler reacting to queue lag. Queue splitting sessions against zero-replica workloads also become ready immediately instead of waiting out the target pod readiness timeout. ### 3.194.0 - 2026-08-17 #### Added * Kafka queue splitting decodes plain-protobuf payloads for jq filters. * Keep Kafka splits warm and drain the fallback before teardown. * Operators reporting to the cloud with identity consent now show per-user and per-target usage in the cloud usage report, matching what a self-hosted license server already reports. #### Changed * Branch database pods store data on per-branch PersistentVolumeClaims by default * The operator now distributes outgoing connections between all mirrord-agents used in a mirrord session. #### Fixed * Fixed a race where two agents could be spawned for the same target pod in one session, leaving that pod's traffic unstolen for the rest of the session. * Fixed agent pods that never become ready, for example because they cannot be scheduled or cannot pull their image, staying in the cluster instead of being cleaned up. * Fixed the Kafka splitting sidecar dropping messages produced while the split was repointing the target workload. * Fixed the operator running every trace-level span when `operator.otelTraceExportUrl` is set, regardless of the configured log level. ### 3.193.0 - 2026-08-12 #### Security * The license server now enforces its Content-Security-Policy on dashboard responses instead of only reporting violations, and sends `Strict-Transport-Security` alongside it. #### Added * Added the `cloud.anonymizeData` chart value (default `false`). Operators authenticated with a cloud API key that carries identity consent attach the same identity fields to cloud telemetry that a self-hosted license server receives; setting the value to `true` keeps cloud telemetry anonymized. * Preview replicas on workload clusters. ### 3.192.0 - 2026-08-10 #### Security * Added a report-only `Content-Security-Policy` to the management dashboard and the hosted web services, along with `Strict-Transport-Security` headers where they were missing. The policy is report-only, so violations are surfaced without blocking any existing behaviour; enforcement follows separately. #### Changed * The operator can now resolve `target.path.labels` into every matching pod in a namespace and coordinate an agent for each eligible pod under one mirrord session, enabling incoming traffic interception across multiple workloads that share the configured labels. ### 3.191.0 - 2026-08-07 #### Added * Added configurable replication factor for Kafka split topics, including copying the source topic's. * Queue-splitting property lists and their secrets can now live in the operator's namespace, shared by all target namespaces. * The active sessions API now honors Kubernetes `fieldSelector` queries, so `mirrord session list --key ` filters sessions in the operator instead of the CLI. In a multi-cluster setup the selector is forwarded to the other clusters as well. * The onboarding wizard's Install Operator step now asks what's blocking installation when setup stalls, so common blockers can be addressed faster. #### Changed * Container-flavor DB branch migration jobs inherit the target's env and envFrom, with connection vars redirected to the branch. * Raised the operator's default CPU limit from `200m` to `500m`. Override with `operator.limits.cpu`. #### Fixed * Fixed copy-target splitting not stamping the session key on matched messages. * Fixed sessions being closed while still starting up * Fixed short sessions lingering instead of expiring promptly * Wait for the Redis Pub/Sub subscription to be live before treating a split session as ready, so a message published right after the session starts is no longer dropped. ### 3.190.0 - 2026-08-03 #### Added * Added named profiles to db branch configuration, letting one cluster hold multiple pod baselines per database type. #### Fixed * Fixed CockroachDB branching from multi-region source databases. * Fixed params-mode CockroachDB source connections being forced to sslmode=disable. ### 3.189.0 - 2026-08-01 #### Security * Members holding the read-only organization role can no longer create, rotate, or revoke API keys, or change license, billing, and payment details, by calling the customer dashboard API directly. These operations now require the organization admin role, and the dashboard hides the corresponding controls from read-only members. * Updated the `event-listener` dependency to 5.4.2, which fixes RUSTSEC-2026-0221: earlier versions unconditionally implement `Send` and `Sync` for the stack-allocated listener created by the `listener!` macro, so a `!Send` tag type could be moved across threads. #### Fixed * Stolen requests are no longer delayed by roughly 40 milliseconds each. The operator API server left Nagle's algorithm enabled on the connections it accepts, so the second of the two writes it makes per stolen request waited for the first to be acknowledged, and that acknowledgement was held by the peer's delayed acknowledgement timer. Only the first request on a connection escaped the wait. Connections accepted by the API server now set `TCP_NODELAY`. * The license server now fails immediately with the underlying error when it cannot bind its configured address, instead of exiting after a delay with a misleading background task error. The address it binds is logged. ### 3.188.0 - 2026-07-30 #### Added * Added private CA and mTLS support for CockroachDB branch source copies. * Operator auth error telemetry now reports which license error occurred, so license failures can be diagnosed instead of all reporting as a generic failure. * PostgreSQL branches can recreate source roles with real attributes, memberships, and grants via the `dbPod.roles: full` chart setting. #### Changed * Improved functional logs, adding more information, logs for http, sqs, and other queue types. #### Fixed * Branch databases now fail fast with a clear error when the migration image cannot be pulled. * Fixed database branching failing on older database images by building every branch init binary against glibc 2.17. * Fixed value\_pattern connection sources silently ignored for ConfigMap-backed env vars. ### 3.187.2 - 2026-07-27 #### Fixed * Faster leader failover and fixed a multicluster startup race. ### 3.187.1 - 2026-07-26 #### Fixed * Fixed a bug where the operator would not delete stale copy\_target pods or delete them prematurely. * Fixed an issue where mirrord sessions were sometimes failing with a "session configuration does not match" error. * Preview sessions will no longer leak traffic when their target's underlying pods linger in the `Terminating` and still receive incoming requests. ### 3.187.0 - 2026-07-23 #### Added * DB branch migrations support user-provided images and image-native Flyway. #### Fixed * Added a warning when the `isolatePods` restart strategy is configured for a StatefulSet target, which uses the standard restart strategy instead. * Fixed MySQL/MariaDB branch migrations failing when the branch image creates no remote root user. * Fixed TLS failure in IAM token generation. * Fixed an issue where sessions using both queue splitting and scaledown would instantly fail with `no target pod with new temporary environment was found` error. ### 3.186.1 - 2026-07-22 #### Fixed * Fix multicluster split queue handoff. ### 3.186.0 - 2026-07-21 #### Added * Add CockroachDB database branching support. * Add MariaDB database branching support. * Added IRSA support for AWS RDS branching. * Operator now servces a no-session diagnostic endpoint used by `mirrord diagnose latency` command. * Support a custom image per DB branch via the `image` field in the mirrord config (all db types), restrictable with the per-database `dbPod.allowedImages` Helm value. * `operator.extraEnv` in the mirrord-operator chart now accepts mapped values, making it possible to define environment variables using `valueFrom` to access the Kubernetes downward API. ### 3.185.1 - 2026-07-19 #### Fixed * A queue splitting session whose `jq_filter` exceeds its evaluation time limit is now cancelled with an error, instead of staying alive while silently never matching a message. Applies to every queue type that supports jq filters: SQS, Kafka, GCP Pub/Sub, Azure Service Bus, Redis Pub/Sub, BullMQ, and Temporal. * Idle previews waking on other services' messages. ### 3.185.0 - 2026-07-17 #### Added * Add support for db params with value pattern in preview env. * Added idle mode for preview environments (`feature.preview.idle`). #### Changed * Renamed `feature.preview.idle.timeout_secs` to `sleep_after_secs`. #### Fixed * Fixed `mirrord operator session kill` and `kill-all` not killing multi-cluster sessions, and `operator status` showing an id `kill` rejects. * Fixed operator API requests intermittently failing when running more than one replica, after a restart. * Fixed orphaned Azure Service Bus subscriptions. ### 3.184.0 - 2026-07-14 #### Added * Added testing for INT-380 * Inject `mirrord-key` into all forwarded messages. * It's now possible to create extra objects alongside the operator/license-server helm installations using the `extraObjects` field. * Preview environments can now selectively filter which labels are copied from the target through the `feature.preview.labels.{include,exclude}` options, analogous to our existing `feature.env.{include,exclude}` options. * Redis branch pods can now run TLS-enabled custom images. * Support CronJob and Job database branching targets. #### Fixed * DB branching now reports a clear error when required source database credential environment variables are missing. * Fix `ASB` same subscription name config. * Fixed copy target with an HTTP filter reliably stealing matching requests. ### 3.183.0 - 2026-07-13 #### Security * Added a `Strict-Transport-Security` (HSTS) header to management dashboard responses, so browsers only ever connect over HTTPS and cannot be downgraded to plaintext HTTP. #### Added * Add Multi Cluster support for Preview Env Queue Splitting. * Add `mTLS` support for `Temporal` queue splitting. * Add a retryable agent error variant. * Add support for generic db branching. * Added Redis Pub/Sub support to mirrord subscribe. * Added `feature.preview.secret_mounts` to mount files into a preview pod from a Kubernetes Secret, so sensitive files can be access-controlled via RBAC separately from the session. * Added an operator API to list active mirrord sessions, cluster-wide or scoped to a namespace (`GET /sessions`, `GET /namespaces/{namespace}/sessions`). * Added support for jq filters in Kafka queue splitting (`jq_filter` in the mirrord config). The jq program runs on a JSON representation of each Kafka message (`topic`, `partition`, `offset`, `timestamp`, `key`, `payload`, and `headers`), so sessions can filter on fields inside the message body. Combined with a header filter, a message must match both. Not supported with the Java Kafka client sidecar. * Allow sharing preview environments. * Queue splitting support for mirror mode (`queue_mode: mirror`), delivering matched messages to both the session and the deployed application. #### Changed * Add queue splitting support for mirrord up. #### Fixed * A database branch whose engine is disabled in the operator's configuration is now marked `Failed` with a clear reason, instead of staying `Pending` forever and leaving the CLI to time out waiting for it to become ready. * Fixed `multi-cluster` envoy RBAC probe. * Fixing dangling resources for `ASB`. * Helm chart no longer prints a spurious "Missing license key" warning on install/upgrade when the operator is configured with only a cloud API key (`cloud.apiKey`). ### 3.182.0 - 2026-07-08 #### Added * Add support for ClickHouse. * Add support for Google Spanner. * Added BullMQ support to `mirrord subscribe`. * Added support for RabbitMQ to `mirrord subscribe` * Implement Flyway migrations. * Operator telemetry now reports a stable per-cluster identifier (the UID of the `default` namespace), so events can be grouped by the cluster they originate from. **This requires an additional RBAC permission:** the operator ClusterRole now needs `get` on the `default` namespace. The bundled Helm chart adds this automatically; if you manage the operator's RBAC yourself, grant `get` on the `default` namespace or the identifier will be omitted. * The operator Helm chart now accepts an optional `operator.clusterName`. When set, the operator reports this human-friendly name on its startup telemetry events, so the cluster shows up with a recognizable label (instead of only an opaque identifier) in the license server and the mirrord cloud. * Wired up GCP PubSub queue splitting to `mirrord subscribe`. #### Changed * Dashboard onboarding: revamped the getting-started wizard into a full-screen flow with a required operator step (generate an API key, install the operator, and verify it via usage) ahead of the CLI step, deriving progress from live usage instead of local storage (RFC 0008). #### Fixed * Fixed cluster-scoped queue split sessions (and their temporary broker resources) leaking after a preview environment was stopped. The split session is now tracked by an annotation rather than an owner reference to the namespaced preview session, so the operator cleans it up when the preview session is gone. ### 3.181.0 - 2026-07-05 #### Changed * operator Helm chart: a cloud API key (`cloud.apiKey.key` / `keyRef` / `gsmRef`) now satisfies the credential requirement, so the operator can be installed without any `license.*` source — it obtains the license over the API (RFC 0008). #### Fixed * operator: `helm uninstall` no longer hangs or strands operator CRs. The chart's `pre-delete` cleanup Job now deletes all operator CRs first — letting the still-running operator execute their finalizers, and stripping the finalizers of any stragglers after a bounded grace — and only then deletes the CRDs and the operator. Previously it deleted the CRDs first, which shut the operator down mid-cleanup (watch streams breaking is treated as a critical error) and could leave CRs stuck `Terminating`, blocking the uninstall indefinitely. * operator: on shutdown, clear the session-TTL finalizer from `MirrordClusterSession`s that are already being deleted, so an exiting operator no longer strands them — and the sessions CRD — stuck in `Terminating`. Live sessions keep the finalizer: in an HA deployment the surviving replicas keep honouring the deletion-delay TTL, and live sessions present at uninstall are cleaned up by the chart's `pre-delete` hook. * operator: the queue-splitting workload-splits task now exits on graceful shutdown instead of stalling it for the full 15s limit. The stall kept the terminating leader pod (and its leadership lease) alive, leaving the operator leaderless long enough for reconnecting clients to lose their sessions during a rolling upgrade. ### 3.180.0 - 2026-07-05 #### Added * Added Cloud SQL Auth Proxy support for MySQL database branching. When the target pod reaches a GCP Cloud SQL MySQL instance only through a loopback Auth Proxy sidecar, the branch init container now starts its own proxy to copy schema and data, matching the existing PostgreSQL behavior. * Added support for fetching database branching connection parameters from Google Secret Manager. The branch init container fetches the secret at data-copy time using the target pod's service account (Workload Identity) via Application Default Credentials, so the operator never reads the secret and needs no extra permissions. * Dashboard: a Settings tab to generate, view, and revoke the operator API key (RFC 0008). #### Changed * Updated `kube-rs` to 4.0. The Kubernetes client now falls back to the OS-native trust store via the rustls platform verifier when no CA is configured, instead of bundled webpki roots. #### Fixed * Clean up split resoures when the configuration is deleted. * Preview sessions now expand the `"*"` wildcard in `split_queues`. ### 3.179.1 - 2026-07-03 #### Fixed * Fixed multi-cluster CRD broadcast sync failing with a 409 conflict when a resource on a remote cluster had fields owned by another field manager (e.g. from a past client-side `kubectl apply`). ### 3.179.0 - 2026-07-02 #### Added * Added a list form for the `split_queues` config so the same queue id can be used more than once across different broker types. * Added a queue-splitting status API that serves a live, namespaced view of active split sessions from the operator's in-memory state, surfaced to users through `mirrord queues`. Each session reports its target, owner, and requested filters, plus a status with the lifecycle phase, the concrete queues resolved from the target (topic / consumer group / queue / subscription per broker), and per-pod patched/ready state. On the primary cluster the view also aggregates split sessions from all connected clusters. * Added support for DynamoDB database branching. Configure it under `feature.db_branches` with `type: dynamodb` and enable it with the `operator.dynamodbBranching` Helm option. * Added the `GET /v1/dashboard/deployments` endpoint that powers the deployments view: per-deployment mirrord coverage (Covered / Uncovered / Unmatched), defect attribution to the most recent prior deploy of the same service and env, per-team defect rates, coverage and defect-reduction KPIs, and trend series. Postgres-only. * operator Helm chart: `cloud.apiKey` value (`key` / `keyRef` / `gsmRef`) to configure the operator's cloud API key — the default credential for authenticating to the mirrord cloud and obtaining the license over the API (RFC 0008). #### Fixed * Fixed multi-cluster CRD sync failing with a 409 conflict when the source resource carried the `kubectl.kubernetes.io/last-applied-configuration` annotation from a client-side `kubectl apply`. The broadcast sync now strips it before applying to remote clusters. ### 3.178.0 - 2026-07-01 #### Added * Added a Deployments tab to the dashboard (gated by `deploymentsEnabled`): coverage and defect-reduction KPIs, staging-defect-rate and defects-prevented trends, a per-team defect-rate breakdown, recommended actions, and a per-deployment detail table showing whether each release was tested with mirrord first (Covered / Uncovered / Unmatched). * The mirrord Operator can now authenticate to the mirrord cloud with a dedicated, rotatable, revocable API key, separate from the license. Generate and manage it from the dashboard, then provide it to the chart via the new `cloud.apiKey` value — either inline (`key`), a Kubernetes secret (`keyRef`), or a Google Secret Manager reference (`gsmRef`). The Operator exchanges the key for short-lived tokens to call the cloud. When unset, the Operator keeps authenticating with its license key, so existing installs are unaffected until they opt in. #### Fixed * Azure Service Bus topic splitting no longer fails to start when its ingest subscription or routing rules already exist from a crashed, aborted, or quickly retried prior session; the existing resources are now adopted instead. * Fix sessions receiving messages from unspecified queues. * Fixed DB branching params mode defaults for when host is not passed in * Fixed MYSQL had no defaults previously. ### 3.177.0 - 2026-06-30 #### Added * Added a `dashboard.deploymentsEnabled` Helm option (off by default) that gates the dashboard's deployments view, wiring `DASHBOARD_DEPLOYMENTS_ENABLED` into the license server and requiring `database.kind=postgres`, mirroring the existing adoption view gate. * The PostgreSQL branch initializer now applies `connectionSettings`. #### Fixed * Fix `mirrord operator session kill` suggestion syntax. ### 3.176.0 - 2026-06-30 #### Added * Added a `POST /v1/defects` license-server endpoint that ingests defect events in batches (upsert by id, so a detector can retry without creating duplicates), feeding the deployments dashboard backend. * Added a `POST /v1/deployments` license-server endpoint that records deploy events (upsert by id), the first slice of the deployments dashboard backend. * Added support for Kafka events to `mirrord subscribe` (Kafka Streams not supported). * Added support for SQS to `mirrord subscribe`. #### Changed * Dashboard: the CSV export is now a button in the top header bar (next to settings) instead of only inside the settings panel, and the adoption heatmap shows each team's size next to its name (same font) instead of tucked in the bottom-left corner. * More adoption and usage dashboard refinements: choose the trend-chart interval (daily / weekly / bi-weekly / monthly) per selected range, dismiss suggested actions, hover any status colour to see its meaning and count, the usage tab now says "Service" instead of "Target" and shows each engineer's team, and the ROI calculator pre-fills the developer count from your data. * Updated adoption tab copy (bucket labels and tooltips) to match the design review. #### Fixed * Azure Service Bus topic splitting no longer fails to start when its ingest subscription already exists. * Copy target sessions now apply jq message filters for SQS, GCP Pub/Sub and Azure Service Bus queue splits. * Fixed two dashboard search issues: the Session Activity search now matches users by display name (not just the raw identifier), and search inputs no longer show a duplicate clear button (the browser's native search "x" is suppressed in favor of the component's own). * Queue splitting and other session features now work through ingress proxies (e.g. GKE Connect Gateway) that reject the encoded connect URL query string, by accepting the connect parameters in a header. ### 3.175.0 - 2026-06-29 #### Added * Added a "Never started" list to the adoption dashboard's engineer-activity row and the group detail dialog, surfacing engineers who have never run a mirrord session by name. Previously they only appeared as an aggregate count. * The operator's license key can now be set to a Google Secret Manager secret name using the new `license.keyGsmRef` option on the Helm chart. #### Changed * Polished the adoption dashboard: a single gradual colour scale, shared across the funnel, team heatmap, group comparison and services and ordered worst-to-best left-to-right; reworded the suggested actions; the scope filter now shows the full group-type label; and the header uses a crisp logo in dark mode. * The adoption dashboard API now attributes each service to the teams whose engineers actually used it in a session (derived from operator session events), in addition to the teams it was explicitly assigned to, so a team with active sessions is no longer reported as owning zero services it demonstrably used. #### Fixed * Fixed the adoption trend chart showing less history on the "All" range than on shorter ranges, and the group detail dialog now counts the services a team actually used in sessions rather than only those explicitly assigned to it. * Legacy SQS `MirrordWorkloadQueueRegistry` splits again honor `operator.sqsSplittingLingerTimeout` as their default drain timeout when they do not carry their own. * Preview Environments no longer fail to start when targetting a workload containing multiple containers that expose the same port. ### 3.174.0 - 2026-06-24 #### Changed * Azure Service Bus queue splitting now includes Azure's response body in its REST errors and logs failed entity operations, so a failure like a missing topic is debuggable. A `NotFound` while creating the operator's ingest subscription is now reported as a clear "topic not found" message that points at the topic name and the Service Bus namespace to check, instead of a bare "resource not found" on an internal subscription name. ### 3.173.0 - 2026-06-24 #### Added * Multi-cluster queue splitting now handles the `operator.metalbear.co/skip-mc-sync` label on `MirrordSplitConfig` and `MirrordPropertyList`. When set to `true`, the operator does not broadcast that resource from the primary to the remote clusters, so users who deploy these CRDs to every cluster themselves. #### Fixed * Fixed a batch of adoption dashboard scoping and edge-case issues: the header counts (engineers / services / teams) now reflect the selected scope instead of always showing company totals; the funnel's "vs company average" comparison now appears when scoping to a business unit, not only a single team; the active-users trend now plots zero-activity days instead of dropping them and can no longer read above 100%; the service list no longer shows an empty page after filtering; the heatmap shows a clear message when a scope has no teams; and the user-classification tooltips were corrected, with "going cold" alerts now also flagging long-abandoned teams. * Fixed the adoption dashboard's team comparison chart collapsing back to the top-level group view when you scoped to a single team. Scoping to a leaf team now keeps the chart at its parent level (e.g. "Teams in Engineering") with the selected team outlined, while the rest of the page scopes to that team. The scope filter also no longer shows an empty "No options" dropdown once you reach a team with no sub-groups. * Queue splitting `drainTimeout` works as follows: absent value waits indefinitely for the fallback queues to drain, and `0` tears down immediately so the workload env patch reverts as soon as no sessions remain. ### 3.172.0 - 2026-06-23 #### Added * `GCP Pub/Sub` queue splitting now `ack_deadline_seconds`, `message_retention_seconds`, and `expiration_seconds` keys from a queue's `queueConfig` `MirrordPropertyList`, letting you configure the temporary subscriptions. ### 3.171.0 - 2026-06-22 #### Changed * The adoption dashboard's trend chart now reflects the selected scope: filtering to a team (or any group) shows that group's "% active over time" instead of the org-wide series. #### Fixed * Azure Service Bus queue splitting now authenticates with the operator pod's Azure Workload Identity when the property list sets only `fully_qualified_namespace`. ### 3.170.0 - 2026-06-19 #### Added * Add `/events` endpoint for `mirrord subscribe` (only HTTP and azure events supported). * Added a `url` property to the RabbitMQ `MirrordPropertyList`, allowing connection details (host, port, vhost, username, password, scheme, SASL mechanism) to be supplied as a single AMQP URI. Individually set properties still take precedence over values parsed from the URL. * Added a deployment-level option to rewrite how user identifiers are displayed on the utilization dashboard (for example, stripping an identity-provider prefix such as `keycloak:`), configured via the chart rather than in-app, and stopped surfacing raw identifiers (such as certificate fingerprints) next to resolved user names. * Added configurable `pg_dump` and `mysqldump` arguments for PostgreSQL and MySQL database branches. #### Changed * Migrating `Kafka` queue splitting to the unified `operator-queue-splitting` crate and `CRDs`. * Migrating `RabbitMQ` queue splitting to the unified `operator-queue-splitting` crate and `CRDs`. * mirrord admin dashboard: addressed product review of the adoption view — consistent active-left bucket order across both charts, swapped the distribution and per-group sections, moved bucket definitions into hover tooltips with a compact legend, consistent section title at any hierarchy depth, and clearer wording on the current-totals note. * mirrord admin dashboard: group types in the directory upload are now free-form labels. The dashboard derives the hierarchy from the uploaded relationships and shows each org their own vocabulary instead of the previously fixed bu / cio\_domain / team set. #### Fixed * Fix edge case where old active `CRDs` with the `*` selctor were not migrated correctly. * Fixed PostgreSQL branch creation failing when the source schema references roles missing on the branch. The source roles are now recreated as empty placeholders before the schema is restored. * mirrord admin dashboard: fixed the directory join diagnostic suggesting display names instead of kubernetes usernames, added template downloads to the adoption onboarding checklist, clarified team-scoped engineer stats on service views, and several small copy and accessibility fixes. ### 3.169.0 - 2026-06-16 #### Added * Added a `valuePattern` field to queue splitting `appConfig` references. #### Changed * GCP Pub/Sub errors now include the attempted operation and target resource, so permission failures report which call was denied instead of a generic `PERMISSION_DENIED`. * Migrating `SQS` to the unified `CRDs` and unfiiying more logic between queue splitting services. * Utilization dashboard's SAML proxy now uses an image from ghcr.io/metalbear.co. * mirrord admin dashboard: the adoption view is now off by default and opt-in via the `dashboard.adoptionEnabled` chart value (it requires the Postgres backend). When off, the Adoption tab and its directory/service-ownership upload sections in Settings are hidden, so deployments on other backends no longer surface adoption UI that only returns errors. The frontend learns whether the feature is on via a new GET /api/v1/dashboard/config endpoint. #### Fixed * Fixed two HTTP 500s in the utilization report (GET /api/v1/reports/usage): one when a user had mirrord sessions with no recorded duration (null aggregate), and one when the selected time range had equal start and end (division by zero). The per-user query now counts all sessions, treats missing durations as zero time, and guards the daily-average division. Also replaced the dev-only "kubectl proxy" error hint shown when the dashboard fails to load. ### 3.168.0 - 2026-06-14 #### Added * Add `POST /v1/dashboard/directory` to upload a directory snapshot, returning an import report with per-reference skip reasons. * Add support for `Redis Pub/Sub`. * Add support for redis with `location: remote`. * Add the adoption tab to the admin dashboard: org tree scoping, per-group adoption funnel, trend, team heatmap, champions, and per-service usage, all driven by the directory and operator-session data. * Added an option to specify a fallback string for the Kafka topic name when the expected environment variable does not exist in the workload. * Added support for splitting [Temporal](https://temporal.io/) task queues (experimental/unstable). * Temporal queue splitting can now filter on headers, search attributes, memo, and input for both activity and workflow tasks. `jq_filter` runs against a decoded task document exposing these fields (input is under `.input`, and a `task_type` field marks `activity` vs `workflow`), and `message_filter` supports `header.` and search attribute keys. * The license server now supports AWS IAM authentication for RDS database connections, enabled via the `DATABASE_AWS_RDS_AUTH` environment variable (`.Values.database.auth.awsIam=true` via Helm). * mirrord-license-server chart: optional Cloud SQL Auth Proxy sidecar (`database.cloudSqlProxy`) for connecting to a GCP Cloud SQL Postgres instance with IAM authentication. #### Fixed * Fix path for `temporal` emulator for `mc`. * Fixed an issue where the operator was not able to run queue splitting with Kafka clusters configured to use namespace-local broker names. ### 3.167.0 - 2026-06-11 #### Added * A new `mirrord-operator-ci` role has been added, which can be used to manage permissions for machine sessions (e.g: `mirrord ci` and `mirrord preview`) in CI runners. * Add an "Export all data" action under Settings in the admin dashboard that downloads the full license-server history as a ZIP of CSV files: usage metrics, users, targets, session activity, CI, friction action items, operator errors, trends, and the adoption directory. * Added `tls.useExistingSecret` to the operator Helm chart, which mounts the secret named by `tls.secret` without the chart creating it (e.g. when it's provisioned by an ExternalSecret). * Azure Service Bus topic queue splitting now uses native Service Bus subscription rules, so topics are split without creating a temporary topic.. * DB Branching with PostgreSQL now supports copying data from PostgreSQL database over cloud-sql-proxy * PostgreSQL database branching now supports source instances reached through a Cloud SQL Auth Proxy sidecar: when the target pod runs the proxy and `gcp_cloud_sql` IAM auth is configured, the branch init container starts its own proxy so it can load data from the instance. The init proxy inherits the target proxy's private-IP setting, so private-only instances work as well. * Preview Environments now fail to start when the deployed pods or their containers fail, crash or otherwise raise an error, such as failing to pull the image, instead of silently succeeding to create the session without reporting that something went wrong. * The admin dashboard now surfaces adoption friction. Connection attempts the operator refuses (policy denials, missing or unauthorized targets, disabled features, license errors, ...) are reported as ranked action items showing the reason, the specific policy/feature/target, and how many distinct users each one blocks, so an admin can see exactly what to fix to help their team adopt. * The operator Helm chart now supports `agent.extraEnv` for adding environment variables to agent pods. #### Fixed * Fixed OpenShift SCC template using hardcoded namespace and service account name instead of Helm release values, causing agent pods to be rejected by SCC when the operator is installed in a namespace other than "mirrord" or with a custom service account name. * Fixed a postgres query failure when getting active CI sessions. * Fixed the admin dashboard All Time Metrics table showing a user as `unknown` when their sessions carried no client username; it now falls back to the user's Kubernetes identity instead of letting the `unknown` placeholder mask it. * Fixed the dashboard SAML proxy not forwarding the authenticated user to the license server: `X-Remote-User` was sourced from `REMOTE_USER`, which `mod_auth_mellon` does not expose as a CGI env var, so the header was never set and the dashboard could not load any data after a successful SAML login. It is now sourced from the `MELLON_NAME_ID` env var the module does export. * When doing Kafka splitting set `min.insync.replicas=1` on created ephemeral topic. * `MirrordKafkaTopicsConsumer` now requires exactly one of `groupIdSources` or `applicationIdSources` to be set, returning a validation error instead of an internal server error when both are absent. ### 3.166.0 - 2026-06-03 #### Added * Added an option to migrate the data from local sqlite to postgress. * Include the client's mirrord CLI version in the Session Start functional log. * Preview environments can now control the number of pod replicas in a session through the new `feature.preview.replicas` option. #### Fixed * Fixed `value_pattern` regex not being applied when the init container connects to the source database. * Kafka splitting no longer stops forwarding messages between sessions. The forwarder stays subscribed as long as the split is alive. ### 3.165.0 - 2026-05-31 #### Added * Added support for seqpacket outgoing connections. * Added support for using an existing ServiceAccount in the mirrord Operator Helm chart. * Preview environments now can be temporarily "paused" by a local `mirrord exec` session configured to use the same target + key as the preview session. While the local session is active incoming traffic won't be directed torwards the preview session and the preview pod won't exist, so no outgoing traffic is produced either. When the local session ends, the preview environment is automatically unpaused. * Preview environments now survive operator restarts, means that your active preview sessions will survive operator upgrades. Note that the session's TTL is still respected even across restarts. #### Changed * Env var to control accepting client certs so you can ignore certs when stealing TLS traffic. #### Fixed * Fixed a transient error during queue splitting where patched resource owner reference update error would cause the split to not work. * Reduced the log noise produced by the preview environment seat controller on `preview stop` command. ### 3.164.0 - 2026-05-27 #### Added * Adds support for optional parsing of S3 event notifications in SQS/SNS messages. S3 object metadata is exposed to jq filters via S3Metadata. #### Fixed * Branch database pods now always use a static mock password instead of the source database password. * Fixed branch pod password mismatch when password source is a Secret without `env_var_name`. * GCP Pub/Sub queue splitting now copies source subscription and topic settings onto temporary resources instead of using Pub/Sub defaults. * On licenses using `KubernetesUser` seat tracking, copy-target and multi-cluster sessions now consume seats by k8s username, matching regular sessions. Previously these paths keyed seats on the client certificate, so a single user across ephemeral containers or multiple machines consumed multiple seats and reported as multiple users in analytics. ### 3.163.0 - 2026-05-22 #### Added * Added support to license-server with multiple replicas. (Including support for postgres database) * Added total count and session duration metrics for CI and preview. * Preview environments are now resilient to pod crashes and evictions. * Preview environments now support mounting user-supplied files in the preview pod via a new `spec.config_mounts` field on `PreviewSession`. #### Changed * Preview sessions are now deleted automatically when their TTL expires, and failed sessions are cleaned up automatically after the retention window configured by `operator.preview.cleanupAfterMins` in the chart configuration. Alongside this change, `mirrord preview status` now only shows active sessions by default, and a new `--failed` flag lets you inspect failed sessions that haven't been cleaned up yet. #### Fixed * Fixed an issue where the operator was printing bogus error logs related to listing PreviewSession resources. * Kafka splitting no longer fails with `MessageSizeTooLarge` when the original topic has a `max.message.bytes` limit above 1 MB. The forwarder producer's client-side cap is lifted, and ephemeral (fallback and session) topics now inherit `max.message.bytes` from the original topic. ### 3.162.0 - 2026-05-19 #### Added * Add `iam` auth for `mysql`. * Add support for unified queue splitting `azure` + `gcp` for preview env. * Added `authType: aks` for multi-cluster authentication using Azure Workload Identity. The operator can now authenticate to remote `AKS` clusters. #### Fixed * Fixed db branching session env overrides when all connection params use literal values or secret sources. The overrides incorrectly returned the source database credentials instead of the branch pod's connection details, causing the app to connect to the source database instead of the branch. ### 3.161.0 - 2026-05-16 #### Added * Add support for `Azure Servicebus` queue splitting. * The operator now listens on IPv6 by default for both the kube API/webhook port and the Prometheus metrics endpoint, with an automatic IPv4 fallback when the primary bind fails. Configurable via `OPERATOR_FALLBACK_ADDR` and `OPERATOR_METRICS_FALLBACK_ADDR`. #### Changed * Add the max concurrent CI session count within a time frame to the license server usage report. * Changes how we display the user name from `identifier` (an id like `jx73h371ha9`) to `client_username` (`meowjesty`, when this name is available). #### Fixed * Added missing `branchcredentials` create permission to the `mirrord-operator-user` ClusterRole. * Check if the ci finalizer has been removed before logging the ci\_controller cleanup BUG message. * Fixed a bug where the configured keepalive period of unused mirrord-agents was not respected. ### 3.160.0 - 2026-05-11 #### Security * Set container-level security contexts for the operator Deployment to run as non-root and drop all Linux capabilities. #### Changed * Update `injectSessionKeyHeader` doc comment to mention HTTP responses and SQS messages. #### Fixed * Dashboard now respects the license seat tracking mode when displaying user identifiers, instead of always preferring the Kubernetes username. ### 3.159.0 - 2026-05-10 #### Added * Surface preview environments alongside normal sessions in `MirrordOperator.status.sessions` so existing clients (`mirrord ls`, `mirrord ui`, browser extension) list them without a separate code path. #### Fixed * Fixed URL path formatting in MetalBear API client. * Kafka splitting issue with multiple sessions split the same topics. * The `APIService` no longer flips to `FailedDiscoveryCheck` after the cluster's front-proxy `CA` rotates. The operator now reloads its `TLS` client verifier when `kube-system/extension-apiserver-authentication` changes. * kafka auth on MSK/AWS with pod identity failing after some time ### 3.158.0 - 2026-04-30 #### Added * Added support for GCP Pub Sub. * Support for single cluster sessions in Multi Cluster. #### Changed * Cluster-wide Secret permissions for db branching are now gated behind `operator.dbBranchingLiteralCredentials` (defaults to `true`). #### Fixed * Fixed `mirrord dump` failing in multi-cluster setups with 2+ workload clusters due to duplicate protocol responses from the multi-cluster router. * Fixed queue splitting timeout on StatefulSet targets when `IsolatePods` restart strategy is enabled. StatefulSets now always use the `Standard` restart strategy, since `IsolatePods` relies on orphaning pods from their ReplicaSet, which does not apply to StatefulSets. * Update RMQ splitting logic so if there are 2 entries with the same name but different cluster then both clusters will have a split queue and forwarder up. * When splitting multiple Kafka topics on the same workload, the target pod now restarts only once instead of once per topic. All topics share a single `MirrordClusterWorkloadPatchRequest`. * kafka ignore transient auth errors ### 3.157.2 - 2026-04-28 No significant changes. ### 3.157.1 - 2026-04-24 No significant changes. ### 3.157.0 - 2026-04-24 #### Added * Support for composite environment variables (with value\_pattern regex) and multi source connection parameters in db branching. * Support for pure value credentials in db branching. #### Fixed * DB branch credentials from a literal value or Secret reference now override the local env vars correctly, even when the target pod does not set them. * Leftover sessions for clusters that do not exist in the registry no longer crash the operator during cleanup. ### 3.156.0 - 2026-04-19 #### Changed * Bump mirrord version to 2.203.1 * Removed the 15 minute MAX limit on database branch `ttl_secs`. ### 3.155.0 - 2026-04-16 #### Changed * Adapt text of operator to use mirrord sessions * Preview environments now ignore the config option `feature.network.incoming.http_filter.ports` to prevent accidentally stealing traffic without a filter. This means that all HTTP filters now unconditionally apply to all intercepted ports. * Replaced the purple accent line under the operator dashboard's AppBar with a subtle top highlight and a theme-appropriate bottom border, matching the direction in the session monitor. #### Fixed * Fixed MySQL branch database init failing on older MySQL images (e.g. 5.7) by building `mysql-branch-init` against glibc 2.17. * Fixed SQS splitting in multi-cluster failing when the queue filter uses a wildcard (`*`). * Fixed bogus error logs printed when preview environments are not disabled. ### 3.154.0 - 2026-04-09 #### Changed * Preview environments no longer override the labels/annotations from the target's pod spec. As a side effect of this change, pods created by preview environments will never be in the "Ready" state, this is intentional. * mirrord-agent pods and ephemeral container specs now use both `command` and `args` instead of just `command`. This is to enable allowlisting with [WorkloadAllowlists](https://docs.cloud.google.com/kubernetes-engine/docs/reference/crds/workloadallowlist). ### 3.153.0 - 2026-04-07 #### Added * Branch database status now includes the created pod name. * Preview env support for Multi Cluster. #### Fixed * ArgoCD does not like empty array values so we remove them when generating the schema. * Preview environments can now target StatefulSets that contain `volumeMount`s in their spec. * Removed default values in chart templates that makes ArgoCD sync failure * Value in helm chart of tls.apiService.insecureSkipTLSVerify can cause perpetual diff in ArgoCD sync fixed * some Helm compatability issues resolved ### 3.152.0 - 2026-03-31 #### Added * Added Support for RabbitMQ splitting. This includes new `queueType` for `MirrordWorkloadQueueRegistry` resource and new `MirrordPropertyList` resource to save connection and queue properties. ### 3.151.2 - 2026-03-29 No significant changes. ### 3.151.1 - 2026-03-29 No significant changes. ### 3.151.0 - 2026-03-28 #### Added * Added operator errors reporting. Error events retention period can be configured with `.server.retention.operatorErrors` (defaults to 30 days). Error reports are accessible at `/api/v1/reports/errors`. * Preview environments now support DB branching and queue splitting. * Preview environments now support `feature.network.incoming.ignore_ports`. * Preview environments now support environment variable manipulation (`feature.env`). * Support for Copy Target for Multi Cluster. #### Changed * OpenAPI schema is generated once and serves an etag to allow caching * Removed API group and resource pairs that don't exist from cluster role template. #### Fixed * Fixed a minor issue where `license.key` is read for CI counting purpose even for regular sessions. * Fixed incorrect YAML indentation for extraVolumes in the mirrord-operator deployment template. * Fixed protobuf openapi endpoint failing due to schema change introduced. ### 3.150.0 - 2026-03-26 #### Added * Operator now sends agent spawn errors in telemetry. * Support setting labels/annotations on preview pods using operator.preview\.labels and operator.preview\.annotations in the chart. ### 3.149.0 - 2026-03-21 #### Added * Dashboard tables now have resizable columns using @metalbear/ui. * Add support for JQ-powered header filters * Preview environment's usage is now collected through our telemetry system * Preview environments now respect the cluster's incoming traffic policies. * Session keys will now be injected into HTTP responses as well (in addition to requests). #### Changed * Add mirrord for ci when license-server is not set up, by using the metalbear backend. * Added a CI flag in operator session event to differentiate regular sessions from CI sessions. As a result, CI runners who are identified by MIRRORD\_CI\_API\_KEY will not be counted as active users. * In SQS, the session key header will now be called `mirrord-key` (instead of `X-Mirrord-Key`) to match injected header names in regular HTTP requests/responses and HTTP conventions. * Preview environment's incoming traffic router is now more robust, reutilizing mirrord's existing `intproxy` logic instead of reimplementing it from scratch. #### Fixed * Fixed detection of message attributes in FIFO SNS envelopes. ### 3.148.2 - 2026-03-10 #### Fixed * Fixed issues with Kafka splitting using the librdkafka client. ### 3.148.1 - 2026-03-10 #### Fixed * MySQL branch init fails when tables have foreign key dependencies. ### 3.148.0 - 2026-03-09 #### Added * Added integration with the Java Kafka sidecar. ### 3.147.0 - 2026-03-05 #### Changed * Make db branching type to be optional, now it always checks for env and env\_from. #### Fixed * Fix env\_from for db params. * Fixed some bogus warning and error logs. ### 3.146.0 - 2026-03-04 #### Added * SQS splitting now supports message attribute injection using the existing key feature. ### 3.145.0 - 2026-03-02 #### Added * You can now update the configuration and/or image of an existing preview environment session by re-running `preview start` with the same key, target, image registry and image repository. * The `feature.preview.ttl_mins`/`--ttl` setting for preview environments now accepts the `"infinite"` string value, which makes the session live indefinitely until being manually stopped. #### Fixed * Fixed an issue where session message processing could get stuck, leaving a persistent port lock that could not be released without restarting the mirrord operator. ### 3.144.0 - 2026-02-26 #### Added * Support for jq filters for SQS splitting. #### Changed * Provide better guidance on what to do on port lock conflict ### 3.143.1 - 2026-02-24 #### Fixed * Fixed a issue where operator telemetry sender strip license server address. * Fixed preview environment's "request was dropped" error when intercepting incoming traffic. ### 3.143.0 - 2026-02-24 #### Added * Add multi cluster session to the CLI status. * The `mirrord preview status` command will now show the remaining TTL of each preview environment session. #### Fixed * Update reqwest dependency - should solve trusting custom TLS if user provides ### 3.142.0 - 2026-02-19 #### Added * Added more `debug` and `warn` logs when resolving librdkafka configuration for Kafka splitting. * Added operator admin dashboard for visualizing utilization metrics. * DB Branching pods limits are now configurable. Aligned all limits to be same for all db branching pods as baseline. * Added feature for injecting `Mirrord-Session-Key` headers into HTTP requests. * mirrord now supports "Preview Environments" - a new type of mirrord session that lives directly in the cluster and can be shared with other users. #### Changed * Lowered log level for `TooManyRequests` errors originating from Kubernetes watch streams. Such errors are quite common at operator startup. * Use proper LicenseKey rather than license hash for ci routes. #### Fixed * When targeting a StatefulSet with `copy_target`, the copy will now correctly mount volumes from PersistentVolumeClaims, fixing failed Pod creation. ### 3.141.0 - 2026-02-16 #### Added * Added `format=json` option to `GET /api/v1/reports/usage` endpoint for admin dashboard metrics. * Operator will now suspend flux resources when there's an active workload patch that rescales its target. #### Fixed * Child session can become stale and never cleaned when using multi cluster. * Multi Cluster sessions were never cleaned when they Fail. * Single cluster session was always checking for parent session before it was created. ### 3.140.1 - 2026-02-12 #### Fixed * Changed duration key to `session_duration` in Session End log as documented. * Fixed bogus warning on OTEL propagation. * Reduced per-session memory usage. * Added IAM auth for Multi Cluster. ### 3.140.0 - 2026-02-09 No significant changes. ### Added * Support for new multi-cluster sessions. ### 3.139.0 - 2026-02-06 #### Changed * Add concurrent ci sessions count to `mirrord operator status`. * operator now emits as user\_id the hashed k8s user (k8s user + salt of org id) to analytics #### Fixed * Fixed an issue that json log config flag is ignored. * Fixed an issue where the operator was using wrong images for branch pod init containers. ### 3.138.0 - 2026-01-30 #### Added * Added agent metrics (currently only for bypassed requests) to the operator metrics. * The operator now propagates OTel context (`baggage`, `traceparent` headers) in http requests. Also, traces and logs can now be exported to URLs given in the operator environment (`OTLP_LOGS_URL`, `OTLP_TRACES_URL`). #### Changed * SQS error logs that originate in AWS SQS service errors now include the original error. ### 3.137.0 - 2026-01-23 #### Added * Add support for db branches IAM. * Added CI cert verification bypass feature in operator status feature list. * Db Branching support for MongoDB. * SQS splitting sessions are now recovered transparently, without restarting the target workload again. #### Fixed * Added test that checks scaledown feature of `copy_target` * Changed leader election logic to drop leadership only after all background tasks are done. * Fixed potential database corruption at license server shutdown. * Outgoing network connections will now only request reverse DNS lookups to the agent when the user has configured hostname-based outgoing network policies. This prevents reverse DNS lookup failures from spamming the logs when they aren't even required in the first place. * Reduced log spam from `operator_session::ci_controller`. ### 3.136.0 - 2026-01-13 #### Changed * Allow bypassing user credentials check when the session is started from mirrord ci start. #### Fixed * Fixed `Session` verbs exposed in operator's `APIResourceList` - replaced `list` with `deletecollection`. * Fixed an issue where `kubectl get v1.operator.metalbear.co.targets` could hang indefinitely in some Kubernetes setups. * Fixed staled copy target state cleanup. ### 3.135.1 - 2026-01-06 No significant changes. ### 3.135.0 - 2026-01-06 #### Added * Support for limiting outgoing connections from mirrord sessions using a kubernetes policy has been added to the operator. * We now log a warning every time a license with an expired issuer is loaded. ### 3.134.0 - 2025-12-24 #### Added * Added `envFrom` as a new type of connection source in DB branching. #### Changed * Enable iptables cleanup on startup by default (instead of erroring out if there are leftover rules). #### Fixed * Fixed an issue with operator's APIService availability with multiple replicas. ### 3.133.0 - 2025-12-21 #### Added * Added the `CiController` to the operator (handles mirrord sessions started with mirrord ci start). Also added ci session handling to the backend (license-server sql now has a new table for bookkeeping of mirrord sessions started from ci, and routes that interact with this table). #### Fixed * fix openapi v2/v3 spec leading to terraform failing ### 3.132.1 - 2025-12-10 #### Fixed * The docker image will now correctly print the license upon startup ### 3.132.0 - 2025-12-10 #### Added * Added security context config for agent pod. [#agent-config.security-context](https://github.com/metalbear-co/operator/issues/agent-config.security-context) * SQS splitting sessions can now be recovered and resumed after operator restart. * Support for the protocol messages introduced in the newest mirrord version has been added, which enable using the `ftruncate` and `futimens` libc functions. #### Fixed * Ignore `fchown` and `fchmod` requests from the layer for security reasons. * SQS messages were filtered wrong when the body was JSON and SNS support was enabled. ### 3.131.0 - 2025-12-04 #### Added * Added PostgreSQL branching. * Added a functional log in license server after each session ends. #### Changed * Changed operator image to be distroless. * The operator now always uses mutating webhooks for patching targets. #### Fixed * Fixed a bug where operator was reconciling unused `MirrordClusterSession`s too rarely. ### 3.130.0 - 2025-11-19 #### Added * The user report generated by the `/v1/reports/usage` endpoint can now be aggregated by a custom key, specified by the new `primary_key` parameter. Check the [docs](https://metalbear.com/mirrord/docs/managing-mirrord/license-server#getting-a-utilisation-report-from-the-license-server) to learn more. ### 3.129.2 - 2025-11-13 #### Added * Added readiness probe in MySQL branch pod. ### 3.129.1 - 2025-11-05 #### Fixed * Fixed a bug where the operator was overloading the Kubernetes API server with PATCH requests for `MirrordClusterSessions`. ### 3.129.0 - 2025-10-29 #### Added * Added multiple data copy modes for bootstrapping a MySQL branch. #### Fixed * Fixed a bug in the `scaledown` feature. * Operator status not showing data for SQS split queues. ### 3.128.0 - 2025-10-21 #### Added * Added Kafka splitting info to the operator status. #### Changed * API calls to MetalBear's servers (e.g: fetching the operator's license) now have a timeout, avoiding infinite hangs when the requests can't complete because of network settings. * Improved logs for lingering temporary SQS queues. * Updated the SQS empty-queue check, according to the SQS docs. #### Fixed * Fixed a bogus error log related to `SingleWatchTask`. ### 3.127.1 - 2025-10-09 #### Changed * Improved pod stability check done during the `isolatePods` restart. Statuses of pod's containers are now taken into account. ### 3.127.0 - 2025-10-06 #### Removed * Removed the legacy method of exposing user session statistics with the `mirrordoperatoruser` resource. Usage can be tracked with the license server. #### Added * Added a warning when an agent is slow to spawn (taking longer than 5 seconds). #### Changed * Adjusted log levels for various reconcilation logs produced by the operator. * Changed the way how the operator creates empty MySQL database if a database name is given by the user. The operator now spawns the branch DB pod with a init container which creates the empty database. The new approach works around potential network policy that limits cross-namespace connections. #### Fixed * Fix `OperatorSessionInfo::hostname` to return the actual hostname. ### 3.126.0 - 2025-09-16 #### Added * Added support for using an HTTP filter in traffic mirroring. #### Changed * Create an empty database using the name in db brnaching configuration. * `MirrordSqsSession` resources now have 4 random letters in their name. #### Fixed * Fixed an issue where the operator was not detecting conflicts between identical `any_of` HTTP filters. * `MirrordSqsSession` resources that failed to start were not being deleted. ### 3.125.0 - 2025-09-11 #### Added * Add option to disable waiting for patched pods when starting queue splitting sessions. A new global operator configuration option `queue_splitting_wait_for_ready_target` has been added to control whether the operator waits for patched pods to become ready when starting Kafka/SQS splitting sessions. This option defaults to `true` (maintaining existing behavior) but can be set to `false` to speed up session start time in environments where pod readiness waiting is not necessary or causes delays due to cluster conditions. Configuration: * Environment variable: `OPERATOR_QUEUE_SPLITTING_WAIT_FOR_READY_TARGET` * Command line flag: `--queue-splitting-wait-for-ready-target` * Default: `true` (waits for ready pods) * Report SQS splitting metrics to prometheus. * The license server now logs the loaded license at startup. #### Changed * Changed MySQL database branch's maximum TTL to 15 minutes. #### Fixed * Fixed a bogus error log related to the `TargetWatch` task. * Fixed a bug in the ArgoCD Rollout restart logic. * Improved the formatting for license server usage reports, including tab names and number display formats. ### 3.124.0 - 2025-09-02 #### Added * Add mysql branch database CR controller and support for using mysql branches in connections to targets and copy targets. * Created an endpoint at `/api/v1/reports/usage` in the license server to provide mirrord usage reports in .xlsx format. * This version of the operator respects the new `workloadRestartTimeout` field in `MirrordWorkloadQueueRegistry`, that controls the timeout for the target workload to restart on the first SQS session start of a target. #### Fixed * Fixed a bug where SQS sessions were failing with a `background task responsible for target monitoring is dead` error. * Fixed a bug where sessions were sometimes failing with a port lock conflict after reconnecting to the operator. ### 3.123.1 - 2025-09-01 #### Changed * Removed the session ID from labels of Prometheus counters. #### Fixed * Fixed a compatability issue with openapi v2 for operator CRDs. ### 3.123.0 - 2025-08-27 #### Changed * When the operator fails to find any applicable `MirrordWorkloadQueueRegistry`, the returned error contains names of all registries found in the session namespace. #### Fixed * Fixed labels and buckets of the `mirrord_operator_ping_latency` Prometheus metric. ### 3.122.0 - 2025-08-22 #### Added * Added rtt metrics (prometheus) for checking mirrord and operator latency. * Add metrics for stolen connections and stolen requests. * Added `OPERATOR_SQS_SPLITTING_LINGER_TIMEOUT` configuration option, that allows for controlling how long (in millis) SQS splits can be kept alive when there are no more SQS sessions, but the are unconsumed unfiltered messages. #### Fixed * Fixed a bug where the operator was not handling stolen TLS messages correctly. * The operator now returns more descriptive errors when it fails to spawn pod agents. ### 3.121.0 - 2025-08-13 #### Added * Added support for blocking `scale_down` when using `copy_target`. ### 3.120.0 - 2025-08-08 #### Added * Ignore configured containers and init containers when using the `copy_target` feature. ### 3.119.2 - 2025-08-06 #### Changed * Bumped mirrord dependencies version. Default mirrord-agent version is now 3.156.0. ### 3.119.1 - 2025-07-29 #### Changed * The operator now sends telemetry data to MetalBear's analytics server even if a license server is configured (unless telemetry is disabled). #### Fixed * Fixed a bug in the `isolatePods` restart strategy. ### 3.119.0 - 2025-07-29 #### Added * Added support for fetching license files from Google Secret Manager (operator and license server). * Added an option to configure max interval between actions taken during an `isolatePods` restart, with a default value of 60 seconds. This allows the restart to always make progress, even in case of frequent/persistent pod failures. * Added support for Kafka splitting with MSK. #### Changed * Workflow dispatched from mirrord triggers operator e2e tests and waits for it to finish. * All errors from failed SQS operations now contain the name or URL of the queue. #### Fixed * Fixed an issue where the operator was prematurely deleting SQS main output queues, failing pods in the target workload after the end of the session. ### 3.118.1 - 2025-07-22 #### Added * Added more informative logs to the code that calls MetalBear API. #### Fixed * Improved SQS splitting code to remove the transient "registry is dirty" error that was occasionally failing SQS sessions' startup. ### 3.118.0 - 2025-07-16 #### Added * Added a timeout when waiting for target pods with patched environment (SQS session startup). * Added support for filtering SNS messages by `Number` message attributes. #### Fixed * Fixed a bug in the `isolatePods` restart strategy related to Argo Rollouts `rollouts-pod-template-hash` label. ### 3.117.0 - 2025-07-09 #### Added * Added a custom workload restart `IsolatePods` strategy, that can be enabled with `OPERATOR_ISOLATE_PODS_RESTART=true`. With this strategy, the operator will restart patched workloads by isolating workload pods from their replica set one by one. The new strategy can be used with mutating webhooks. * Added support for selecting SQS queues environment variables with regular expressions. * Added support for specifying a catch-all queue id (specified as `*` queue id) to SQS queue splitting. Only for SQS splits currently. ```json5 { "split_queues": { // * means all queues under this target "*": { "queue_type": "SQS", "message_filter": { "x-pg-tenant": "Aviram" } } } } ``` * Added support for using queue `fallback_name` in the MirrordWorkloadQueueRegistry. If source variables doesn't exist, fallback to that name. #### Changed * Added validation when running license-gen to generate a new license (now we check for a bunch of things that would result in errors or warnings in the operator). * Improved errors and warnings produced by the SQS splitting code. * Validate a couple more things in license-gen command to avoid invalid license being created. #### Fixed * Fixed issues with mirrord sessions using both the copy target and and the SQS splitting features. * Improved how the operator handles client requests when no mirrord-agent is alive for the given session. ### 3.116.2 - 2025-07-01 #### Changed * The operator now prepares and creates copied pods in the background, returning an early success response from the create handler. The progress of the copied target is exposed in the resource status. ### 3.116.1 - 2025-06-24 #### Added * Add the default agent image version that the operator will use as a label. #### Changed * Change incorrect license key behavior to not print the used key. * When attempting to report metrics to Jira, levels of log messages emitted on failure are now improved. ### 3.116.0 - 2025-06-17 #### Added * Add session duration reporting when a session ends if a Jira webhook URL has been configured. * Support for getting SQS queue names/urls from env vars that originate in config maps. #### Changed * Update error and code comments about namespaced profile. ### 3.115.0 - 2025-06-10 #### Changed * Bump operator `mirrord` version to `3.144.0`. #### Fixed * The operator now exits without unnecessary delay when it receives SIGTERM. ### 3.114.2 - 2025-06-09 #### Changed * Change `active_users` field in "Max seat count reached!" log to number instead of slice. ### 3.114.1 - 2025-06-04 #### Fixed * Fix issue with scaling where a target pod can remain "unavailble" from operator's perspective even though the target has started and is available. ### 3.114.0 - 2025-05-28 #### Added * Add config `OPERATOR_KAFKA_SPLITTING_TOPIC_FORMAT` to control Kafka splitting topic names. with default value that results in same format as previous versions `mirrord-tmp-{{RANDOM}}{{FALLBACK}}{{ORIGINAL_TOPIC}}` #### Changed * Change operator to always send telemetries when using license server while asserting it's not configured to be our endpoint * The operator now returns 404 response when a requested copied pod is not found. ### 3.113.1 - 2025-05-16 #### Fixed * Fixed an issue where the operator was not exposing the OpenAPI V2 schema. ### 3.113.0 - 2025-05-14 #### Added * Add option to toggle our `/openapi/v2` with `OPERATOR_OPENAPI_V2_ENABLE` environment variable, this can be useful if there is some incopetability issue when operator is installed. (hashicorp's terraform module can experiance issues with openapi v2 enabled) ### 3.112.0 - 2025-05-12 #### Added * Add alternative tracking method for license seats where the kuberentes username will be the counter. #### Changed * Renamed CRD `MirrordProfile` to `MirrordClusterProfile`. ### 3.111.0 - 2025-05-06 #### Added * `runAsUser` field in copy-target containers can be configured by operator admin. #### Changed * Adds a reflector for caching mirrord policies. #### Fixed * Fixed an issue were Kafka splitting sessions without copy target could fail on target readiness timeout. ### 3.110.0 - 2025-04-28 #### Added * Added `OPERATOR_LICENSE_KEY` and `OPERATOR_LICENSE_PATH` values to operator-license-server (equivalent to `LICENSE_KEY` and `LICENSE_PATH`) * Added an option to configure TTL for idle Kafka splits. #### Changed * SQS filters are now case insensitive by default. #### Fixed * Stopping SQS queue splitters faster at the end of a splitting session. ### 3.109.1 - 2025-04-17 #### Fixed * Fixed a bug where agents were getting OOM killed due a to traffic redirection loop. ### 3.109.0 - 2025-04-15 #### Added * Add initial e2e tests for license-server. * Add new flag named `OPERATOR_LICENSE_ALLOW_SEAT_OVERAGES` that allows enterpirse licenses to go past defined max seats (enabled by default) * Added support for Kafka splitting without copying the session target. #### Changed * Names of temporary Kafka topics created during Kafka splitting now end with the original topic name. * Policies can now be applied to copy-targets (based on the original target). #### Fixed * Kafka splitting sessions now fail early when the operator is unable to create temporary topics. * Put rdkafka compilation behind a feature to fix CI cmake issue. * More correct dates for fetch of active-users to use the "last billing date" as from date not just "last 30 days". * The operator now correctly removes the `operator.metalbear.co/patched` annotation from target workloads that are no longer patched. ### 3.108.0 - 2025-03-29 #### Added * Support for passing queue URL instead of queue name for SQS queue splitting. * The operator now enforces mirrord profile selection, based on `spec.requireProfile` and `spec.profileAllowlist` fields of mirrord policies. #### Fixed * Fixed an issue where the operator was loosing connections to agents that were producing large quantities of messages. * Reintroduce the openapi v2 endoint after fix verification. ### 3.107.2 - 2025-03-19 #### Changed * Disable `/openapi/v2` route due to critical breakage. #### Fixed * Remove pod-template-hash from copied pod (fixes copy-target when targeting pods). ### 3.107.1 - 2025-03-17 #### Fixed * Fixed a bug where operator was not able to revert scaledown of Rollouts using `workloadRef`. * Small fix to license-server telemetry endpoint. ### 3.107.0 - 2025-03-16 #### Fixed * Session Error status was not being set, and users got a copy-target timeout instead of the actual SQS error. * Add support for `/openapi/v2` with addition of several fixes to the v3 variant. * Fix ArgoCD warning and potentially auto sync disable not working ### 3.106.1 - 2025-03-03 #### Removed * Removed per-pod `SubjectAccessReview`s done in the operator during multi-pod sessions. ### 3.106.0 - 2025-02-24 #### Added * Added support for filtered stealing of HTTPS requests. #### Fixed * Added a 30 second timeout for fetching topics metadata from a Kafka cluster. * Agent and copy-target pod controllers no longer print unnecessary warnings when deleting a pod fails with 404. ### 3.105.1 - 2025-02-19 #### Fixed * Fixed issues with Rollout targets without a label selector. ### 3.105.0 - 2025-02-06 #### Added * Added support for enforcing patterns on HTTP header filters. * Added a configuration option to set the port on which the agents accept operator connections. The port number can be set with the `OPERATOR_AGENT_PORT` environment variable. * Added support for Kubernetes ReplicaSet targets. #### Fixed * Fixed a bug where the operator was not able to communicate with the agents over TLS. ### 3.104.2 - 2025-01-28 #### Fixed * Fixed how the operator negotiates `mirrord-protocol` version with the client during targeted sessions. * `agent.privileged` no longer affects targetless agents' pods. ### 3.104.1 - 2025-01-27 #### Added * Operator now detects conflicts between port subscriptions with HTTP filters. A confict occurs when an active subscription uses an HTTP filter that matches a superset of requests matched by the HTTP filter used by the new subscription request. #### Fixed * use correct mirrord dependencies ### 3.104.0 - 2025-01-27 #### Added * Support for `rmdir`, `statfs`, `unlink` and `unlinkat` file requests. * Support for targeting a Kubernetes `Service`. #### Fixed * Operator now performs a subject access review on each pod targeted in a user session. * Operator now correctly sends session termination reason to the client. * Operator no longer fails to start a targetless session when the agent config specifies the ephemeral container agent mode. The operator now always spawns pod agents for targetless sessions. ### 3.103.0 - 2025-01-14 #### Added * Added policy to control file ops. * Added policy to exclude env vars. #### Changed * Updated DataDog dashboard to include list of users in last 30 days. ### 3.102.0 - 2025-01-02 #### Added * Allow users to use the original image for copy target pod by specifying a flag. * Added support for `mkdir` and `mkdirat` hooks. #### Changed * Added new trusted license issuer valid until January 2027. #### Fixed * The operator now issues a warning when fetched `MirrordPolicy`/`MirrordClusterPolicy` contains an unknown blocked feature. ### 3.101.0 - 2024-12-12 #### Added * Added Prometheus endpoint with `/metrics` route and enabled with `OPERATOR_METRICS_ENABLED` environment variable * `mirrord_license_valid_seconds` - seconds left for current license validity * `mirrord_sessions_create_total` - count of created sessions, labels: `client_hostname`, `client_name`, `client_user` and `user_id` * `mirrord_sessions_duration` - histogram for session durations after they are ended, labels: `client_hostname`, `client_name`, `client_user` and `user_id` * Added support for `MirrordClusterPolicy`. #### Fixed * Operator no longer changes target container name when copying target. ### 3.100.0 - 2024-12-10 #### Added * Added SQS to `mirrord operator status` reporting. * Added support for `SASL_SSL` security protocol in Kafka splitting. * Added support for blocking `mirror` feature in `MirrordPolicy`. * Allow Kafka properties to be set from the optional `.spec.loadFromSecret` field in `MirrordKafkaClientConfig`. By default, the operator can only read secrets in the operator's namespace (`mirrord` by default). #### Changed * Update dependencies. #### Fixed * Fixed a bug where target workload scale was not being restored after operator restart (`copy_target` + `scale_down`). * Added "transaction" like behavour so ArgoCD `Application` will unpatch if any error had occured while patching the train. * Eliminated SQS-operator panic when a container in the target workload does not have environment variables. ### 3.99.0 - 2024-11-20 #### Added * Added support for ArgoCD Application auto-sync pausing when mirrord is used with `scale_down` enabled. ### 3.98.1 - 2024-11-08 #### Changed * Bump dependencies \[mirrord agent version] ### 3.98.0 - 2024-11-06 #### Added * Exposed user session statistics, accessible with `kubectl get mirrordoperatorusers -o json`. #### Changed * Adds a new match for EOF on websocket. Plus some error logs that should help showing other potential websocket issues. * Bump dependencies * Kafka splitting component now evaluates message filters in order of session start. If two user filters match a certain message, this message is received by the user that started their mirrord session first. ### 3.97.0 - 2024-10-17 #### Added * Added a configuration option for timeout when a session has no available pod targets. * Added Kafka splitting components. #### Changed * Bumped dependencies. * Updated rust version. ### 3.96.0 - 2024-10-13 #### Added * Added a configuration option for timeout when a session has no available pod targets. * Added Kafka splitting components. #### Changed * Bump dependencies ### 3.95.3 - 2024-10-07 #### Changed * Refactor ResolvedTarget (and many of its checks) to mirrord. Now the GET target here is on its way to being deprecated. #### Fixed * Failed and completed agent pods are now correctly removed by the operator. ### 3.95.2 - 2024-09-24 #### Fixed * Fixed operator unable to handle requests with too much headers ### 3.95.1 - 2024-09-12 No significant changes. ### 3.95.0 - 2024-09-11 #### Added * More logs from the SQS-splitting operator and improved existing logs. #### Fixed * Fixed `Operator Monthly Users` status not counting users that were last seen on the current day. [#mau](https://github.com/metalbear-co/operator/issues/mau) * Update so `SubjectAccessReviewSpec` values that are derived from header fields are properly unescaped ### 3.94.0 - 2024-09-03 #### Added * Added support for using multiple HTTP filters in one session. #### Fixed * Operator container now fails when the default client CA cannot be fetched. * Update operator to not use Impersonation API. Now there is another request to check whether the user should be allowed via `SubjectAccessReview`. ### 3.93.0 - 2024-08-21 #### Added * Added `ReadDirBatch` support in the operator. ### 3.92.1 - 2024-08-18 #### Fixed * incompatibility with older clients [metalbear-co/mirrord#2675](https://github.com/metalbear-co/mirrord/issues/2675) ### 3.92.0 - 2024-08-17 #### Added * Split SQS queues between multiple mirrord sessions/users. ### 3.91.0 - 2024-08-14 #### Fixed * Update Datadog Mirrord Operator Dashboard to group and disply by user\_id and not client\_name. * Fixed a bug where initial license fetch failure would stop the operator from refreshing the license. ### 3.90.0 - 2024-07-31 #### Added * Adds new env var OPERATOR\_SESSION\_MAX\_TIME allowing the user to put a time limit on mirrord sessions. #### Fixed * Fixed a bug where targetless agents were not terminating after finished mirrord session. ### 3.89.0 - 2024-07-31 #### Changed * Allows targeting StatefulSet without the copy\_target feature, both as an immediate target, and as a workload ref in targeted rollouts. ### 3.88.1 - 2024-07-18 #### Fixed * Handle log messages ### 3.88.0 - 2024-07-17 #### Fixed * Operator API now lists all targetable resources for mirrord CLI versions > \=3.109.0. ### 3.87.0 - 2024-07-12 #### Changed * Introduce enforcement of client certificates when accessing operator api's other than `/health`. * Changed loglevel of instrument license verify to trace #### Fixed * Fixed a bug where multi-pod sessions where targeting copied pods. Removed copied pods from target list returned by the operator. * Fixed bogus error log that was often printed during mirrord sessions. * Removed some unnecessary error logs. ### 3.86.0 - 2024-07-02 #### Added * Functional logs now contain unique client id produced from client's certificate. * Added support for streamed HTTP responses. ### 3.85.0 - 2024-06-25 #### Added * Operator now adjusts to changes in the target state that occur during the mirrord session (e.g. scaling targeted deployments/rollouts, targeted container restarts). * Added support for streamed HTTP requests. #### Fixed * Operator now properly handles case where mirrord CLI uses a higher version of mirrord protocol. * `ImagePullSecrets` on copied pods now respect secrets specified in the agent config. * Fixed panic in a background task responsible for watching agent config file. ### 3.84.0 - 2024-06-18 #### Added * Add support for CronJob and StatefulSets with copy target ### 3.83.0 - 2024-06-12 #### Removed * Removed `pause` feature that is no longer supported in new agent versions. #### Added * Add hot-reload for agent config file. #### Fixed * Fix handling rollout with malformed resources spec ### 3.82.0 - 2024-06-02 #### Added * Adds Job as a possible target when copy\_target feature is enabled (checked in mirrord). * Add hot-reload for license file. #### Changed * Use the labels and annotations from agent\_config when creating an agent pod. * `port-locks` subresource is deprecated. The endpoint is left for compatibility, but returns nothing. #### Fixed * Make CopyTarget feature use only pods without jobs. ### 3.81.0 - 2024-05-26 #### Changed * Bump mirrord agent version to have new agent json\_log config. * Begin enforcing `max_seats` and syncing usage with cloud on online licenses. * Update dependecies #### Fixed * Refactor how the operator stores the active license to be a watched task. Now each part that access the license for information holds a receiver, instead of a copy of the license, and has access to the updated license. * Fixed agent version check performed when using agent TLS connections. ### 3.80.0 - 2024-04-28 #### Added * Operator now passes the TLS cert to the agent, allowing for authentication. * Add example of DataDog dashboard for operator. ([integrations/datadog/Mirrord\_Operator\_Dashboard.json](https://github.com/metalbear-co/docs-changelog/tree/main/integrations/datadog/Mirrord_Operator_Dashboard.json)) #### Changed * Dropped using `Job` for agents and started spawing `Pods` directly. * Normalized events for important events. * Copy Target * Port Steal * Port Mirror * Port Release * Session Start * Session End | field | description | events | | ---------------------------------------------------------- | --------------------------------------------------- | ----------------------------------------- | | client\_hostname | `whoami::hostname` of client | `All` | | client\_name | `whoami::realname` of client | `All` | | client\_user | kubenetes user of client (via k8s RBAC) | `All` | | http\_filter | http filter for stealing http connections | `Port Steal` | | port | port number (if relevant) | `Port Steal` `Port Mirror` `Port Release` | | scale\_down | if target was scaled down | `Copy Target` | | session\_id | unique id for individial mirrord execution (base64) | `Port Steal` | | `Port Mirror` `Port Release` `Session Start` `Session End` | | | | session\_duration | time in seconds of session's existance | `Session End` | | target | kubernetes resource targeted | `All` | * Bump mirrord dependencies to 3.99.0 * On telemetry send error, don't drop all events - just truncate #### Fixed * Fixed DAU/MAU counters not being reset * Fixed the issue where stealing/mirroring from multi-pod targets would result in duplicate locks in operator status. ### 3.79.0 - 2024-04-14 #### Added * Add option to output logs in JSON format using `OPERATOR_JSON_LOG` = true * Added an option to connect with the agents using TLS. ### 3.78.1 - 2024-04-10 #### Changed * Make copy job always run as user 1000 * Run operator as user to have better permissions control #### Fixed * Reworked port locking: system-wide `PortLocks` holds all active port locks, layer-scoped `LayerPortLocks` owns port locks made by specific `LayerHandler`. Removed separate port locks info from the operator session. * License validation now properly handles `4xx` responses from the licensing server. Such responses are now treated as non-transient and block usage. * Copy job now sets terminationGracePeriodSeconds to 1 to fix cleanup issues ### 3.78.0 - 2024-04-03 #### Changed * Use the new agent config for agent-image + Bump mirrord version. ### 3.77.1 - 2024-03-06 #### Fixed * Re-add the OperatorSession event on end of session. ### 3.77.0 - 2024-03-03 #### Added * Adds operator's session management API exposed through kube. The new routes allows the user to `kill_all`, `kill_one` or `retain_active` sessions, and are (currently) available through `mirrord operator session` cli commands. #### Fixed * Enable telemetry even if the user set OPERATOR\_TELEMETRY\_ENABLED to false (for teams licenses). * For copy-target feature changed so creation of `pod_template` is done with impersonated client before copying said target to check the user is allowed to copy the resource. ### 3.76.2 - 2024-02-27 #### Changed * Updated dependencies ### 3.76.1 - 2024-02-27 #### Changed * Updated dependencies ### 3.76.0 - 2024-02-21 #### Added * Added support for Argo rollouts in `copy_target` feature. #### Fixed * Remove always returning weird answer for any unmatching request, resolving issue when kube asks for OpenAPIv2 schema and we just say here take garbage.. * Fixed a bug where duplicate port locks where displayed in operator status when targeting multi-pod targets. ### 3.75.1 - 2024-02-13 #### Fixed * Fixed filtered steal port lock being displayed as mirror in operator status. * Don't impersonate for copy job ### 3.75.0 - 2024-01-24 #### Added * Report port locks and filters in operator status #### Changed * Make telemetry optional only on Enterprise, failing if no communication is done in max time (1h). * Use the same operator certificate dates (valid\_from, valid\_to) for user certificates (the ones signed by the operator). * User id is now produced from user certificate public key instead of fingerprint. #### Fixed * Show local name and hostname in mirrord operator status * Sidecars and init containers are now copied for deployments in copy feature ### 3.74.0 - 2024-01-16 #### Added * Extended certificate extension with additional data, exposed license generation methods, added license id to telemetry events #### Fixed * Increased max headers from 100 to 1000 ### 3.73.0 - 2024-01-14 #### Added * Add namespace to the session status * Adds a LicenseExtension type that we insert into our certificates as an extension to distinguish between Teams and Enterprise. #### Changed * Do not allow a teams license to be added using the local (certificate) file flow (reserved for enterprise license). * Include copy targets in operator status. * license\_server url is now defined at compile-time only (either comptime env var, or default value). #### Fixed * Fixed copy job idle timeout logic - the job now terminates immediately when the agent is unloaded. [#copy-job-ttl](https://github.com/metalbear-co/operator/issues/copy-job-ttl) ### 3.72.0 - 2024-01-08 #### Added * The operator checks the policies that applies to each run and blocks port subscription if forbidden by a policy. #### Changed * Listing includes container targets for resources with multiple containers. ### 3.71.0 - 2024-01-05 #### Added * Added support for reading agent config from file so we can use configmap #### Changed * Check if the license has been issued by us (also allows mock values via env var OPERATOR\_LICENSE\_ISSUER\_PUBLIC\_KEY). Refactored the initialization of OperatorLicense to take this into account. * Change logging/tracing to only log on new/exit of functions instead of each enter/leave to reduce logs ### 3.70.0 - 2024-01-01 #### Changed * Operator now can run without certificate, falling back to creating self-signed on runtime ### 3.69.1 - 2023-12-26 #### Changed * Do not deny route access if the license check failed due to a network error. Implement From for converting between \&LicenseError and TelemetryEvent. #### Fixed * Fixed format of error responses. Errors are now handled properly by the `kube` create in the CLI. * Add copying of init containers and side containers in copy feature * Create copy target in configured default namespace (not kube's default namespace) if not specified. * Fixed intervals of license checks. ### 3.69.0 - 2023-12-19 #### Added * Added license authorization check (via [axum middleware](https://docs.rs/axum/latest/axum/middleware/index.html)) to block unlicensed/expired licenses from accessing certain functionalities in the operator. ### 3.68.1 - 2023-11-30 #### Fixed * Fixed connectionid reuse when deployment with multiple pods ### 3.68.0 - 2023-11-08 #### Added * Added a `scale down` feature to the `copy target` feature. * Added support for container selection * Add initial implementation for `CopyTargetCrd`. ### 3.67.0 - 2023-09-18 No significant changes. ### 3.65.2 - 2023-09-11 #### Changed * When in targetless mode always use non ephemeral containers. ### 3.64.2 - 2023-09-05 No significant changes. ### 3.63.0 - 2023-08-28 No significant changes. ### 3.62.0 - 2023-08-27 No significant changes. ### 3.61.0 - 2023-08-22 No significant changes. ### 3.60.0 - 2023-08-21 #### Added * Add handling for OutOfPods error where nodes are too full. ### 3.59.0 - 2023-08-18 No significant changes. ### 3.58.0 - 2023-08-17 No significant changes. ### 3.57.1 - 2023-08-15 #### Added * Add Towncrier. * Add startup event and alive event --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation by asking a question. Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters: ``` GET https://metalbear.com/mirrord/docs/changelog/operator.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.