How monday.com Runs Agent Evals Against Real Dependencies: Webinar Recap

An agent eval suite’s outcome can only be trustworthy if it’s operating in an environment similar to production. You can have the best grading logic in the world, but if the agent is calling mocked databases and fake APIs, you’re not testing how it behaves in the real world, you’re testing how it behaves in the mocks you built. That gap is what we dug into on September 15, when we sat down with Dor Cohen, AI Engineering Director at monday.com, for an interview and demo on how his team architected running agent evals against a live staging cluster instead of mocks.

If you missed the session, here’s the recap: what agent evals actually need to check, why mocks fall short compared to the agent calling real systems, and how monday.com built their agent evals infrastructure.

What are agent evals and why you need them #

Testing an agent isn’t the same as testing a model. An agent is a package: the model, the tools it calls, and the dependencies behind those tools. Unlike a unit test, where you know exactly what output to expect, an agent is non-deterministic, as it can pick a different tool or path on every run. So the eval has to check the full trajectory it takes to get there, not just whether the final answer sounded right.

Dor shared an example of why testing the full chain matters. They had an agent that was asked to retrieve 600 items. It called the right tools, formatted a confident, well-structured answer, and returned a response that read as correct. Except it had only processed about 500 of them. Every individual tool call worked. The final answer looked fine, but the task wasn’t actually done. An eval that only grades the final answer would have passed it but an eval that checks trajectory and completeness catches it.

Where mocks fall short #

One very important part in the agent eval chain we discussed above is the dependencies your agent calls. The default way to run evals is against mocks because they’re lightweight, fast to set up, and isolated, and for a lot of testing that’s exactly what you want. But for an agent that’s calling real databases, services, and APIs in production, mocks introduce three problems.

• They drift: a mock has to be kept in sync with your production environment by hand, and it never is.
• The data isn’t real either, synthetic rows present in mocks are cleaner and thinner than what actually lives in your database, so they can’t expose the edge cases production will.
• And mocks can’t represent state. When an agent reads data and takes a downstream action, a mock either forgets the write happened or fakes it, so you can’t actually tell what the agent left behind.

Staging doesn’t have any of those problems by construction. It stays current with production because it’s the last stop before it. It runs on real data, at close to production scale, with all the variety and mess that comes with that. And because the agent’s actions actually land, you can check the real end state, not a simulated one. Fidelity to production is what makes or breaks the quality of your agent evals.

The objection to running evals against staging (or other pre-prod environments) is reasonable, though: it’s not lightweight, not isolated, and not easy to wire up. That’s the problem monday.com solved with mirrord.

How mirrord fits into the pipeline #

mirrord connects a process running locally, in a CI job, or in an eval runner, to a real Kubernetes cluster, so it behaves as if it’s deployed there: real traffic mirrored or stolen from a live pod, real env vars and secrets imported straight from the target, real reads and writes on the remote filesystem. DB branching and queue splitting isolate each eval session’s writes, so multiple runs, from different engineers or different CI jobs, can hit the same cluster at the same time without stepping on each other’s data.

There’s no separate deploy step as well, you just wrap your existing eval command and run it with mirrord:

mirrord exec --target deployment/orders-api \
    -- pytest evals/agent_suite.py

And in CI you can just use mirrord ci start instead of mirrord exec, with the same arguments. When a model, a prompt, or a tool changes, your CI job runs the suite against real dependencies before the change reaches a merge gate.

Inside monday.com’s agent evals pipeline #

monday.com runs 700+ microservices behind its agents, with the usual sprawl that comes with that scale: authentication, feature flags, permission layers, and a long list of third-party integrations. Mocking these dependencies or setting them up each time in an ephemeral environment would’ve been a huge investment for them in terms of both time and money. Their estimate for a dedicated evaluation environment came out to about a month of setup, plus ongoing maintenance across 700+ services just to keep it in sync. Obviously, they ended up going the more practical route of pointing their evals at staging with mirrord, which their developers were already using for local development.

The difference shows up in the day-to-day now: agent eval suites run in minutes instead of hours, the added infrastructure cost is close to zero since monday.com already pays for staging, and they get an output that they can actually trust.

Catching a regression before it ships #

monday.com treats a model upgrade the same way it treats any other version change to the agent: it runs the full eval suite against the new model before switching. When the team tested Sonnet 5 against their existing default model, goal completion dropped by about 10%, and agent correctness dropped by about the same. The suite failed to cross the pass threshold.

This was great for the team as it gave them a clear signal that the upgrade wasn’t safe to ship as-is, pointed to specific failure points in tool selection and prompt adherence, and gave them a concrete direction for what to adjust before trying again. Without a suite running against real dependencies, this is the kind of regression that tends to surface only after a customer hits it.

Three ways monday.com actually runs AI agent evals #

Dor shared that their eval pipeline isn’t just limited to CI pipelines, it’s the same underlying infrastructure triggered from three different places, all producing consistent results through mirrord and LangSmith.

1. From Claude Code, a developer working locally can invoke an offline evals skill right after making a change. It asks for the eval scope, all cases or a specific category, connects to dependencies living in staging through mirrord, and runs the suite. Results land in LangSmith with input, reference output, actual output, and a score per metric.
2. From Slack, monday.com has an internal Claude-based evaluation agent, nicknamed “Eevee,” that can be asked to run evals with specific parameters, like swapping in a different model for a test run. It uses the same infrastructure, produces the same LangSmith output, and posts a summary back in Slack. This is actually how the team caught the ~10% drop in goal completion during the Sonnet 5 test.
3. From CI, eval results show up alongside regular checks on the pull request, each with a score and a mode: observed, which is informational, or blocked, which prevents the merge, depending on the severity of the change and the product it’s touching.

monday.com runs these “offline evals” alongside “online” ones. Offline evals run against their staging environment with controlled datasets, gating changes before they ship. Online evals grade live agent behavior continuously after it is shipped to production. When something surfaces in production that the offline dataset didn’t cover, it gets pulled into that dataset for future regression testing, closing the loop between what ships and what gets tested next.

Metrics to track for agent evals #

monday.com’s base eval pipelines track six metrics:

1. Goal completion: did the agent finish the whole task, not just appear to
2. Agent correctness: was the answer factually and logically right
3. Tool precision: did it pick the correct tools for the task and call them in the correct sequence
4. Trajectory: was the path taken the expected one
5. Groundedness: are the agent’s claims backed by evidence it actually had
6. Intercompleteness: did it gather everything it needed before answering

Some of these are fully deterministic, like whether the required tool got called. Others need an LLM-as-judge, for things like whether the agent understood the goal or stayed in scope. And individual product teams at monday.com add their own domain-specific evaluators on top of the base set, because what counts as a passing score depends entirely on the product and the use case.

Takeaways #

A few things worth carrying away from the session:

• Agent evals should test more than just the final output: tools picked, dependencies used, trajectory taken, and completeness all matter.
• Mocks drift, and synthetic test data can’t replicate the variety and edge cases that live in production.
• Most teams already have the right environment for running agent evals properly. Staging is current with production by definition. Building a dedicated eval environment from scratch is usually more expensive and slower than pointing your suite at the one you already run.
• Online and offline evals reinforce each other. Edge cases from production belong in your offline dataset, so the two tracks compound over time instead of drifting apart.

If you’d like to see the full webinar recording including the demo Dor showed, you can get it here. And if you want to set up a 1:1 session with one of our engineers who’ll walk you through how to connect your agent evals to real dependencies with mirrord, you can book a demo with us.