Run behavioral eval cases against a live agent.

A case is a YAML file: a prompt sent to the agent as the logged-in member, and deterministic checks run against the reply (contains, regex, verbatim lines from a reference file).

A case may also seed workspace files before the prompt (seed:), send a workspace action as a page would and check its result (action:), and check workspace files after the turn (files:), so a suite asserts on what the agent made and not only on what it said.

Cases live in the owner's agent repo (e.g. /evals/*.yaml), versioned next to the behavior they test — and never synced to the agent.

Every case runs from a cleared context by default: a context marker is sent before each prompt so the case measures the configuration, not the suite's own accumulated history (later cases in a long run otherwise inherit the tone and facts of earlier answers).

Pass --clear-between=false for a suite whose cases are meant to build on one another.

Each case declares the role it is written for (owner or user).

Run owner cases logged in as yourself; for user-role cases either switch to a test-user login (mutiro auth switch, --role user), or probe one of the agent's real user conversations with --in-conversation-with: the probe is sent as you (owner interjection), but the engine answering stays bound to the conversation's own user, so the case exercises that user self — its workspace, its tools, its context.

Examples:

Run the owner-role cases from an agent repo

mutiro agent evals run ./my-agent/evals --agent my-agent

Run user-role cases while logged in as a test user

mutiro agent evals run ./my-agent/evals --agent my-agent --role user

Run user-role cases against the agent's email self

mutiro agent evals run ./my-agent/evals --agent my-agent --role user --in-conversation-with agentmail

Run a single case by name

mutiro agent evals run ./my-agent/evals --agent my-agent --only quote-email-template