> For the complete documentation index, see [llms.txt](https://metalbear.com/mirrord/docs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://metalbear.com/mirrord/docs/sharing-the-cluster/queue-splitting/bullmq.md). # BullMQ This page covers queue splitting for [BullMQ](https://bullmq.io/), a Redis-backed persistent job queue. For the general concepts and the message filter reference shared by all queue services, see the [Queue Splitting overview](https://metalbear.com/mirrord/docs/sharing-the-cluster/queue-splitting.md). The word "queue" on this page refers to a BullMQ queue (backed by Redis lists, hashes, and sorted sets under the `bull::*` key prefix). {% hint style="info" %} Queue splitting for BullMQ requires mirrord operator `3.175.0` or later and mirrord CLI `3.223.0` or later. {% endhint %} ## How It Works ![BullMQ queue splitting flow](https://2838204941-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FghNlSpMkqkYKZCZQsHRt%2Fuploads%2Fgit-blob-1a6b77d7f4cf8e642c841e2283467fc58f6bf3de%2Fbullmq.svg?alt=media\&token=58449c82-576b-42fa-8d42-f704a72c0861) BullMQ stores jobs as Redis data structures: a wait list (`bull::wait`), per-job hashes (`bull::`), and an ID counter (`bull::id`). The mirrord operator's BullMQ forwarder dequeues jobs from the original wait list using `BLPOP`, inspects the job's `data` field (which must be valid JSON), and re-enqueues each job to the appropriate temporary queue based on the users' filters. When the first mirrord BullMQ splitting session starts, the operator creates a temporary "main" queue for the deployed workload and patches the workload's queue-name environment variable to read from it. The forwarder then begins dequeuing from the original queue: jobs whose JSON data matches the user's filter are re-enqueued to a temporary per-session queue that the local application reads, and everything else is re-enqueued to the main queue. If a second user starts a session on the same queue, the operator creates another per-session queue and adds the second user's filter to the routing logic. If the filters defined by two users both match some job, one of the users will receive the job at random. When a session ends, the operator deletes the per-session queue's Redis keys. When all sessions end, the operator restores the workload to read from the original queue and deletes the main queue's Redis keys. If the main queue still holds jobs that the workload hasn't finished, such as jobs that wait, are delayed, or that a worker holds, the operator keeps the main queue rather than delete those jobs, and deletes it once no job in it is unfinished. The `Drained` condition in the status of the `MirrordSplitConfig` names each queue that was kept this way. If the queue's `MirrordPropertyList` is gone by then, for example with the target's namespace, the operator can't reach Redis any more, and you must delete the queue's keys yourself. ## Enabling BullMQ Splitting in Your Cluster {% stepper %} {% step %} #### Enable BullMQ splitting in the Helm chart Enable the `operator.bullmqSplitting` setting in the [mirrord-operator Helm chart](https://github.com/metalbear-co/charts/blob/main/mirrord-operator/values.yaml). {% endstep %} {% step %} #### Create a MirrordPropertyList The operator needs to connect to your Redis instance to dequeue and re-enqueue jobs. Define the connection in a `MirrordPropertyList` ([`CustomResource`](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/)) in the same namespace as the target workload (and the `MirrordSplitConfig`), or in the operator's namespace to share it across namespaces - see [Sharing Property Lists Across Namespaces](https://metalbear.com/mirrord/docs/sharing-the-cluster/queue-splitting.md#sharing-property-lists-across-namespaces). ```yaml apiVersion: mirrord.metalbear.co/v1 kind: MirrordPropertyList metadata: name: bullmq-config namespace: workers spec: properties: - name: url value: redis://redis.redis-ns.svc.cluster.local:6379 ``` Supported properties: | Property | Description | Required | Default | | ------------- | :------------------------------------------------------------------: | :--------------------: | :-----: | | `url` | Full Redis URL, e.g. `redis://host:6379/0`. Used as-is when present. | One of `url` or `host` | | | `host` | Redis hostname, used when `url` is absent. | One of `url` or `host` | | | `port` | Redis port, used with `host`. | No | `6379` | | `password` | Password. | No | | | `tls` | Set to `"true"` to connect over TLS (`rediss://`). | No | `false` | | `db` | Database index. | No | `0` | | {% endstep %} | | | | {% step %} #### Create a MirrordSplitConfig On operator installation with `operator.bullmqSplitting` enabled, a new [`CustomResource`](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/) type is defined in your cluster - `MirrordSplitConfig`. Users with permissions to get CRDs can verify its existence with `kubectl get crd mirrordsplitconfigs.queues.mirrord.metalbear.co`. Create a `MirrordSplitConfig` for the target workload. BullMQ uses `kind: bullMq` in queue entries. ```yaml apiVersion: queues.mirrord.metalbear.co/v1 kind: MirrordSplitConfig metadata: name: worker-split namespace: workers spec: targetRef: apiVersion: apps/v1 kind: Deployment name: order-worker clientConfigs: bullMq: bullmq-config queues: - id: orders kind: bullMq appConfig: queue: - env: BULLMQ_QUEUE ``` The `MirrordSplitConfig` above says that: 1. It targets the deployment `order-worker` in namespace `workers`. 2. The Redis connection comes from the `bullmq-config` `MirrordPropertyList`. 3. The deployment consumes one BullMQ queue, whose name is in environment variable `BULLMQ_QUEUE`. 4. The queue can be referenced in a mirrord config under ID `orders`. #### Link the config to the deployed consumer The `MirrordSplitConfig` is a namespaced resource. The target workload reference is specified with `spec.targetRef`: * `apiVersion` - API version of the Kubernetes workload (e.g. `apps/v1`). * `kind` - type of the workload. Supported: `Deployment`, `StatefulSet`, `Rollout`. * `name` - name of the workload. #### Describe consumed queues Each entry in the `spec.queues` list describes a BullMQ queue consumed by the workload: * `id` - arbitrary queue ID that developers [reference](#setting-a-filter) from their mirrord config. * `kind` - must be `bullMq`. * `clientConfig` (optional) - name of a `MirrordPropertyList` with the Redis connection. Can also be set once for all BullMQ queues with `spec.clientConfigs.bullMq`. * `queueConfig` (optional) - name of a `MirrordPropertyList` with forwarder tuning. Supported properties: * `dequeue_timeout` - how long `BLPOP` waits before retrying, in seconds (default: `5`). * `jq_time_limit` - max milliseconds for jq filter evaluation per job (default: `200`). * `appConfig.queue` - the BullMQ queue name. Uses the same structure as other queue services (`env`, `envLike`, `volume`, `fallback`, `valueSelector`, `valuePattern`, `containerNames`). {% hint style="warning" %} The mirrord operator can only read consumer's environment variables if they are either: 1. defined directly in the workload's pod template, with the value defined in `value` or in `valueFrom` via config map reference; or 2. loaded from config maps using `envFrom`. {% endhint %} {% endstep %} {% endstepper %} ## Drain timeout After the last session against a target ends, the operator waits for the deployed workload to finish the jobs in the main queue, then restores the workload and deletes the temporary queues. It waits for jobs that wait, that are delayed or prioritized, that wait for their children, and that a live worker holds. A job whose worker died without finishing it holds the drain while other workers of the main queue run, and for one `stalledInterval` after the last of them stopped, because a worker puts the job back to wait. Once no worker has run for one `stalledInterval`, for example after an autoscaler scaled the workload to zero, or while the workload's only pod takes longer than that to restart, the job no longer holds the drain, and the operator keeps the main queue with that job instead (see [How it works](#how-it-works)). The drain timeout caps the wait: | Setting | Unit | Scope | Effect | | ----------------------------------------------- | ------- | --------- | ----------------------------------- | | `spec.drainTimeout` on the `MirrordSplitConfig` | seconds | One split | Wins over the cluster-wide default. | | `drainTimeout` | Behavior | | -------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- | | unset (both) | Wait until no job in the main queue holds the drain, as described above. The operator keeps the main queue if a job whose worker died is left in it. | | `0` | Tear down immediately. The operator keeps the main queue if jobs in it are unfinished. | | `N` | Wait up to `N` seconds, then tear down. The operator keeps the main queue if jobs in it are unfinished. | ## Temporary queue names By default, temporary queues are named `mirrord-tmp--` for user queues and `mirrord-tmp--main-` for the queue the patched workload reads. To use a different format, set `spec.tmpNameTemplate` in the `MirrordSplitConfig`: ```yaml spec: tmpNameTemplate: "dev-mirrord-{{RANDOM}}{{FALLBACK}}{{ORIGINAL}}" ``` The template can use these placeholders: * `{{RANDOM}}` (required) - random alphabetic characters, so that every split and session gets queues of its own. A template without it is rejected, because two splits of the same queue would get the same queue names, and cleaning up one split would delete the other's queue. Before you upgrade to an operator that requires it, end the sessions on a config whose template lacks it and let its split end: the upgraded operator can't resume such a split, and leaves the Redis keys of its main queue behind. * `{{FALLBACK}}` - resolves to `-main-` for the queue the patched workload reads or `-` for user queues. * `{{ORIGINAL}}` - name of the original queue being split. ## Setting a filter For the full filter reference (`queue_type`, `message_filter`, `jq_filter`), see the [overview](https://metalbear.com/mirrord/docs/sharing-the-cluster/queue-splitting.md#setting-a-filter-for-a-mirrord-run). BullMQ uses `queue_type: BullMQ`. BullMQ jobs carry their payload in a `data` field stored as a JSON string in the job hash. Filters match against the parsed JSON value of this `data` field. `message_filter` matches regexes against top-level JSON fields; `jq_filter` runs against the full parsed JSON. Filtering on a top-level JSON field: ```json { "operator": true, "target": "deployment/order-worker/container/worker", "feature": { "split_queues": { "orders": { "queue_type": "BullMQ", "message_filter": { "tenant": "^test$" } } } } } ``` In the example above, the local application will receive only jobs whose `data` JSON has a top-level `tenant` field equal to `test`. Filtering on the payload with `jq_filter`: ```json { "operator": true, "target": "deployment/order-worker/container/worker", "feature": { "split_queues": { "orders": { "queue_type": "BullMQ", "jq_filter": ".priority == \"high\"" } } } } ``` In the example above, the local application will receive only jobs whose `data` JSON contains `"priority": "high"`. {% hint style="info" %} Jobs whose `data` field is not valid JSON never match attribute or jq filters, so they are routed to the deployed (cluster) application. {% endhint %} For compatibility with older operator releases and the legacy `containers` spelling, see [Selecting Containers](https://metalbear.com/mirrord/docs/sharing-the-cluster/queue-splitting.md#selecting-containers). --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation by asking a question. Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters: ``` GET https://metalbear.com/mirrord/docs/sharing-the-cluster/queue-splitting/bullmq.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.