> ## Documentation Index
> Fetch the complete documentation index at: https://docs.serval.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Evaluations

> Test how the Help Desk Agent handles realistic scenarios with safe, repeatable evaluation runs

<Info>
  Evaluations are in beta. If you don't see **Evaluations** in your team
  settings, ask your Serval contact to enable it for your organization.
</Info>

Evaluations let you test how the Help Desk Agent handles realistic support scenarios without creating real tickets. Each evaluation runs a saved scenario against your team's current configuration and records the full conversation, every tool and workflow call, and the outcome. Workflow calls and access requests are simulated by default, so nothing changes in your connected systems unless you explicitly allow it.

For example, simulate a new hire asking for Figma access and verify that the agent routes the request for approval, without touching your real systems.

To get started, open your team settings and select **Evaluations**.

<Note>
  Running evaluations requires the **Drafter** team role or above. Simulating a requester other than yourself requires a **Manager** or org **Admin**. See [Permissions](#permissions).
</Note>

## Configs and runs

An evaluation *config* is a saved scenario: the requester, their opening message, the situation, and what a good resolution looks like. Every evaluation runs from a config. Save a scenario once and re-run it any time to see how the agent's behavior changes as your team's guidance, knowledge, and workflows evolve.

Each config keeps a revision history. Editing a config's scenario creates a new revision, and every run records the exact revision it executed, so past results stay comparable even after the scenario changes.

A *run* plays the scenario against the Help Desk Agent on an isolated evaluation ticket. Evaluation tickets never appear in your Help Desk.

<Note>
  Evaluations use only the published revisions of your skills and workflows, the same ones the agent uses on real tickets. Draft changes don't affect an evaluation until you publish them.
</Note>

## Create a config

From **Evaluations**, select **Saved configs**, then **New config**, and define the scenario:

* **Name**: How this config appears in the library. Must be unique per team.
* **Description** (optional): What the scenario exercises.
* **Opening message**: What the requester writes to open the ticket.
* **Requester**: Who the ticket is from. The agent treats the ticket as this person's request, and any tools it calls act on their behalf.
* **Conversation shape**: How the conversation unfolds.
  * **Single reply**: The agent answers the opening message once. Fastest, for a quick check that it responds well.
  * **Auto conversation**: A simulated requester replies based on the scenario background until the scenario is completed or the conversation stalls.
* **Scenario background** (auto conversation only): The requester's real situation and what a genuine fix requires. This is the run's ground truth: the simulated requester replies based on it, and scoring judges the outcome against it. It isn't the agent's first message.
* **Workflow execution** (auto conversation only): Whether the agent's workflow calls are simulated or allowed to run live. See [Simulated and live workflows](#simulated-and-live-workflows).
* **Expected outcome** (optional): What a good resolution looks like. When set, Serval scores each run pass or fail against it.

### Revision history

Saving scenario changes creates a new revision. Renaming a config or changing its description doesn't. When a config has more than one revision, its page shows a **Revision history** section. Open an older revision to run the scenario exactly as it was, or select **Restore as new revision** to make it current again.

Deleting a config removes it from the library. Past runs keep the revision they executed.

## Run an evaluation

Start a run from a config's page. To tweak the scenario first, select **New evaluation** from the **Evaluations** list, pick a config, and adjust any field for a one-off variation. Fields you change are marked **This run**, and a **Reset to config** action puts them back. Overrides apply only to that run and never modify the saved config.

Then choose how it runs:

* **Run as configured**: An autonomous run using the config's conversation shape. An auto conversation ends when the issue is resolved, the agent escalates, or the conversation stalls.
* **I'll respond**: An interactive run where you play the requester, replying turn by turn.

If the run impersonates another user or allows live workflows, Serval asks you to confirm before it starts.

### Interactive runs

In an interactive run, the run page shows the conversation as it happens. While the status is **Waiting for next turn**, type the requester's next message and select **Send reply**. Select **End evaluation** to finish the run and score it. An interactive run left idle for two hours is marked **Stopped** automatically.

## Simulated and live workflows

By default, every workflow and access request the agent calls during an evaluation is simulated. The agent sees a realistic mocked result and behaves as if the call ran, but nothing changes in your connected systems.

To exercise specific workflows for real, set **Workflow execution** to **Allow selected workflows live** and add the workflows. Only the workflows you select may execute with real side effects. Every workflow you don't select remains simulated. Live workflows run with access scoped to the requester, and you confirm the selection again before the evaluation starts.

Keep in mind:

* Adding a workflow to the live selection requires permission to run published workflows (the **Contributor** role or above).
* Only workflows that are deployed and enabled can run live. If a workflow in a saved config no longer qualifies, Serval blocks the run until you remove that workflow from the selection.
* A workflow marked **AI-filled form** asks the requester for input. The simulated requester fills in the form from the scenario before the workflow runs.
* Live calls happen as the conversation progresses. A live workflow that ran in an earlier turn can't be undone, even if you stop the run.

## Results

Open a run from the **Evaluations** list to review it:

* **Status**: **Running the Help Desk Agent** while in progress, **Waiting for next turn** during an interactive run, and then **Completed**, **Failed**, or **Stopped**.
* **Outcome**: **Resolved** or **Not resolved**, with an AI-written summary of what happened and a confidence score.
* **Expected result**: When the run has an expected outcome, a verdict shows whether the run met it. The verdict compares the meaning of the outcome, not its exact wording.
* **Setup**: The config and revision the run executed, its mode, requester, workflows configured to run live, scenario context, and expected result.
* **Conversation**: The full transcript, rendered like a real ticket thread. Every tool and workflow call appears as a card labeled **live** or **mock** with its output. Each turn also shows the skills, workflows, knowledge, and access the agent considered.

Each config's page also lists its recent runs, so you can compare how the same scenario played out across revisions.

## Create a config from a real ticket

To turn a real ticket into a repeatable test case, open the ticket's actions menu and select **Create evaluation config from ticket**. Catalyst reads the ticket, derives the scenario from what happened, and saves it as a config with every workflow simulated.

<Note>
  A ticket with more than one end user can't be turned into a config, because a config simulates exactly one requester.
</Note>

## Ask Catalyst

Catalyst can create, run, and inspect evaluations for you. Ask in plain language, for example "run an evaluation where a new hire asks for Figma access" or "re-run the VPN reset scenario and summarize what changed." Catalyst starts autonomous runs, waits for the results, and opens each run and config it touches so you can follow along. It always asks before allowing a workflow to run live, and it tells you when a run impersonates another user.

## Permissions

| Action                                    | Requires                                                         |
| ----------------------------------------- | ---------------------------------------------------------------- |
| Run evaluations and view runs and configs | **Drafter** team role or above                                   |
| Create, edit, or delete configs           | **Drafter** team role or above                                   |
| Simulate a requester other than yourself  | **Manager** team role or org **Admin**                           |
| Allow workflows to run live               | Permission to run published workflows (**Contributor** or above) |

For more on team roles, see [Permissions](/sections/documentation/platform/permissions).