Simulations

Simulations let you test how the Help Desk Agent handles realistic support scenarios without creating real tickets. Each simulation runs a saved scenario against your team’s current configuration and records the full conversation, every tool and workflow call, and the outcome. Workflow calls and access requests are simulated by default, so nothing changes in your connected systems unless you explicitly allow it. For example, simulate a new hire asking for Figma access and verify that the agent routes the request for approval, without touching your real systems. To get started, open Simulations in your team settings, or in Development where enabled.

​

Browse simulations

Simulations opens your suite library. Open a suite to see its configs, latest run health, and performance history. Use New suite to create a collection of configs that you can run together. Open a suite to browse its configs. Configs that don’t belong to a suite appear under Not in a suite on the Simulations landing page; select Review configs to open their list. Suite and config searches cover the loaded rows; select Load more suites or Load more configs to include additional rows.

​

Configs and runs

A simulation config is a saved scenario: the requester, their opening message, the situation, and what a good resolution looks like. Every simulation runs from a config. Save a scenario once and re-run it any time to see how the agent’s behavior changes as your team’s guidance, knowledge, and workflows evolve. Each config keeps a revision history. Editing a config’s scenario creates a new revision, and every run records the exact revision it executed, so past results stay comparable even after the scenario changes. A run plays the scenario against the Help Desk Agent on an isolated simulation ticket. Simulation tickets never appear in your Help Desk.

​

Create a config

From Simulations, select New config. This requires Catalyst access and opens a blank config editor beside a fresh chat. Fill in the scenario yourself, or describe what you want to test in the chat. Valid edits save to your Catalyst draft automatically. Opening the editor doesn’t send a message or publish a config. The editor organizes the config into:

• Name and Description: Edit these directly at the top of the editor, below the Simulation config banner. They describe how the config appears in the library and what it exercises.
• Simulation details: Choose the Requester, write the Opening message, and provide Scenario context describing their situation and the details they can share. New configs default to a conversation with a simulated requester. Select Run a single reply to stop after the first agent response.
• Workflow configuration: Choose which workflows can run live during a conversation. Single-reply simulations simulate workflows. See Simulated and live workflows.
• Judge criteria: Provide an optional Expected result describing a successful outcome. When set, Serval scores each completed run pass or fail against it.

In Catalyst, the top bar shows the config’s Draft or Published status and its published revision when available. Select Publish when ready, or Propose for review if your team requires approval. Wait for changes to finish saving and complete required fields before taking either action. A review includes the pending change set from the chat, which may contain other resources.

​

Edit a config

Opening a saved config takes you directly to its Catalyst editor beside a fresh chat. Returning to a chat whose changes were published shows the current published config. Change the fields directly to update the staged draft automatically, or ask Catalyst to make changes. Historical comparisons retain the recorded scenario settings. Manual changes saved in Catalyst remain a draft until you publish them. If your team requires approval, select Propose for review in the config tab and publish the change set after approval. Hover over Draft in the top bar and select Compare versions. This read-only view uses the retained draft; newer edits appear after autosave completes. The current Catalyst Draft is available alongside published revisions, with the matching published revision selected for the before-and-after comparison by default. Use Visual to review the fields or Code for the full configuration diff. Comparing doesn’t publish or change the config.

​

Revision history

Publishing scenario changes from Catalyst creates a new revision. Renaming a config or changing its description doesn’t. In Catalyst, hover over the Published badge to see the revision’s publication time and publisher, when recorded. Select Full version history in that popover to see published revisions alongside the editor. Select Restore version on an older version to stage its scenario as a draft, then publish it or propose it for review when ready. Wait for edits to finish saving before restoring. If the chat already has a saved draft, confirm before replacing its scenario. Existing history links also open Catalyst with the history panel visible. When at least two revisions have saved scenarios, select Compare versions in the version popover or history panel. Choose the base and comparison versions, or swap them. Published revisions store scenario settings only, so historical comparisons exclude name and description. Draft comparisons also include name and description when the published-state metadata is available. To delete a published config, open More actions in its editor and select Delete config, then Stage deletion. The config stays available until you publish the change. Past runs keep the revision they executed.

​

Create and edit a suite

From Simulations, select New suite. With Catalyst access, a blank suite editor opens beside a fresh chat. Enter a name and description and add configs, or ask Catalyst to help. Valid edits save to the staged draft automatically; select Publish when ready, or Propose for review if your team requires approval. Opening the editor doesn’t send a message. Open a saved suite and select Edit in Catalyst to change its description or membership. You can review the proposed changes before publishing them. Without Catalyst access, the suite form lets you choose a name, description, and member configs directly. In either editor, configs already assigned to another suite are disabled and show their existing suite. Remove a config from that suite before adding it to another. The picker searches loaded configs; use Load more configs to browse further. You can save an empty suite and add configs later. Suites appear as a flat library, with their configs listed alphabetically. Select Run suite to run its saved configs together. Each run captures the config revisions it uses; later edits don’t change past results. Open a suite run to see its performance summary, individual outcomes, and links to each config’s simulation transcript.

​

Suite dashboard

Each suite page shows Latest run for its most recent finished launch within the latest 25 launches. Expected-result pass rate includes only simulations with a verdict. Expected-result misses, run errors, and unavailable scores remain separate. An active run shows its progress without replacing the last finished run’s health. Performance history plots one pass-rate point per scored, finished launch across that same window. Select a point to open its run. Active launches and launches without scores appear as gaps; loading more entries in Recent suite runs doesn’t change the chart’s window. The Latest run header includes the run’s timestamp, status, and a View run link. Open that run for individual simulation results and captured settings. The suite dashboard focuses on aggregate health and performance history; its config list shows the suite’s current membership.

​

Investigate run results

Run results open in Simulations. With Catalyst access, Open in Catalyst opens the same result beside a chat, including while a run is queued or running and after it passes. Opening the run does not send a message or prepare an investigation prompt. For a finished run with an expected-result miss, an unresolved agent outcome, or an execution error, the action becomes Investigate with Catalyst and prepares an investigation prompt. Review the prompt and send it to begin. If the run is already open in Catalyst, the investigation action prepares the prompt in that chat. Catalyst can inspect the recorded member results, captured settings, transcripts, judge results, and workflow executions. It distinguishes expected escalation from unexpected non-resolution and suggests next steps. Investigating does not edit, publish, or rerun simulations. Missing scores or configs without an expected result do not trigger an investigation preset on their own; Open in Catalyst remains available. Results already shown beside a Catalyst chat omit the redundant open action.

​

Run a simulation

Open a suite and choose a config, or select Review configs under Not in a suite to find a config without a suite. The config opens in its Catalyst editor. Select Run config to open run options for the published revision shown in the editor. Publish any draft changes first. Review the settings, then select Run simulation to start an autonomous run and open its result page in Simulations. In run options, adjust Live workflows for this run or, when version pinning is enabled for your team, use Test specific or unpublished versions to choose workflow and skill versions. You can select a draft or an older published version. Pinning a workflow does not make it run live. Reset to config discards your overrides, and Cancel closes the dialog without starting a run. These changes never update the published config; the result records the effective settings under Run parameters. For an interactive run where you play the requester, open Run options beside Run config and select Run interactively. Choose any overrides, then select Start interactive run. An autonomous conversation ends when the issue is resolved, the agent escalates, or the conversation stalls. To change a scenario, edit and publish the config in Catalyst. For other one-off scenario variations, ask Catalyst to run it with overrides. Opening an old run-setup link takes you to the config editor without starting a run. If the run impersonates another user or allows live workflows, Serval asks you to confirm before it starts.

​

Rerun a past simulation

Open a finished config run from its config’s Run history, then select Rerun on the result page. This starts a new run from the original config revision and saved overrides, including participants, conversation mode, live workflow selections, approval modes, and version pins. Later edits to the config don’t change these settings. An interactive run starts a fresh interactive conversation. Rerunning doesn’t replay the recorded conversation. Resources without version pins use their current published versions. Serval checks your current permissions and confirms impersonation or live workflows again before starting.

​

Interactive runs

In an interactive run, the run page shows the conversation as it happens. While the status is Waiting for next turn, type the requester’s next message and select Send reply. Select End simulation to finish the run and score it. An interactive run left idle for two hours is marked Stopped automatically.

​

Run a suite from a workflow

Workflows can run a suite as one unit and wait for its result without polling. serval.common.runSimulationSuite starts every config in a published suite as a single suite run, pauses the workflow until that run is completed or failed, and returns the recorded run with its per-config verdicts and a passed flag. Use it where a check should block on simulation results, such as a test that must pass before a branch is promoted. A workflow runs with an API key or worker identity, so it does not confirm config revisions the way a person does; the suite’s current published revisions run. Live workflows stay simulated unless the call sets liveWorkflowsConfirmed, and simulating other users still requires the permission to do so.

​

Simulated and live workflows

By default, every workflow and access request the agent calls during a simulation is simulated. The agent sees a realistic mocked result and behaves as if the call ran, but nothing changes in your connected systems. To exercise specific workflows for real, select Add workflow under Live workflows. Workflows in this list may execute with real side effects. Every workflow you don’t add remains simulated; leave the list empty to simulate all workflows. Live workflows run with access scoped to the requester, and you confirm the selection again before the simulation starts. Keep in mind:

• Adding a workflow to the live selection requires permission to run published workflows (the Contributor role or above).
• Only workflows that are deployed and enabled can run live. If a workflow in a saved config no longer qualifies, Serval blocks the run until you remove that workflow from the selection.
• A workflow marked AI-filled form asks the requester for input. The simulated requester fills in the form from the scenario before the workflow runs.
• Live calls happen as the conversation progresses. A live workflow that ran in an earlier turn can’t be undone, even if you stop the run.

​

Results

Open a config’s Run history or a suite’s Recent suite runs to review a run in Simulations. Selecting an individual simulation from a suite opens its result page there too. Open in Catalyst or Investigate with Catalyst is an explicit handoff to chat; both show the same results, captured settings, and transcript beside the conversation. Opening results does not create a Catalyst conversation. For a suite run, ask Catalyst questions such as “Why was one item not resolved?” Catalyst reads the members and config revisions captured for that launch before inspecting an individual transcript. You can also ask for a suite’s recent runs. A Not resolved outcome can still pass the expected-result check, for example when the scenario expects the agent to escalate the issue to IT. The run view groups the result, Judge criteria, Conversation, and Run parameters into sections. A completed run without an expected-result verdict shows No verdict; completion alone does not mean it passed.

• Status: Running agent name while in progress, using your configured help desk agent name; Waiting for next turn during an interactive run; and then Completed, Failed, or Stopped.
• Outcome: Resolved or Not resolved, with an AI-written summary of what happened and a confidence score.
• Expected result: When the run has an expected outcome, a verdict shows whether the run met it. The verdict compares the meaning of the outcome, not its exact wording.
• Run parameters: The config and revision the run executed, its mode, requester and other participants, opening message, scenario context, live workflow selections, and pinned versions. These are the settings captured for the run, so later config edits do not change the result view.
• Conversation: The full transcript, rendered like a real ticket thread. Every tool and workflow call appears as a card labeled live or mock with its output. Each turn also shows the skills, workflows, knowledge, and access the agent considered.

Select Run history beside Run config in a saved config’s Catalyst editor to see its past runs across all revisions. The popup shows execution status, judge results, revision, and timestamp. Use Older and Newer to page through runs; select a run to open its results in Simulations. Run history is available while editing an existing config, and closing it keeps your edits. Past results remain available through direct run links and suite history without Catalyst access; config creation and editing use Catalyst exclusively.

​

Create a config from a real ticket

To turn a real ticket into a repeatable test case, open the ticket’s actions menu and select Create simulation config from ticket. Catalyst reads the ticket, derives the scenario from what happened, and saves it as a config with every workflow simulated.

​

Promote simulations between teams

Where promotions are enabled, saved simulation configs and simulation suites move between linked teams like other configuration. In a promotion’s comparison and review they appear as Simulation config and Simulation suite.

• Participants are carried by user, never by email. A config saved before participants were stored by user is translated when the promotion is prepared; if an address has no account and the organization’s domain policy refuses to create one, that config is left out of the comparison until the participant is fixed on the source team.
• A suite is promoted with its member configs. Every member must be part of the same promotion or already exist on the receiving team; staging a suite without one is refused and the message names the missing simulation config.
• Promotion recreates a member config the receiving team deleted, because the source suite still lists it. Remove the config from the suite on the source team instead.

​

Ask Catalyst

Catalyst can create, edit, run, and inspect simulation configs and suites for you. Ask in plain language, for example “run a simulation where a new hire asks for Figma access” or “re-run the VPN reset scenario and summarize what changed.” Catalyst starts autonomous runs, waits for the results, and opens each run and config it touches so you can follow along. It asks for confirmation before running workflows live or simulating another user. You can start from New config, New suite, or a saved config or suite’s edit action when you have Catalyst access. When a config is open beside the chat, Catalyst receives that config as context for your messages. Runs execute published config revisions. Unpublished simulation config or suite changes in a Catalyst chat are not part of a run; publish them before asking Catalyst to test those changes.

​

Permissions

The libraries, dashboard, and Catalyst editors respect these permissions. Users with Read can inspect saved configs and results; authoring and execution controls require their respective permissions. Publishing also follows the team’s Catalyst review and publication policy. Assign Read alongside Edit or Run when someone needs to browse simulations and results. Edit does not grant permission to execute a scenario. A suite requires permission for every participant in every member config; Serval does not silently skip unauthorized configs. The default Drafter and Contributor roles include Read and Edit, but cannot run simulations. The default Builder and App Builder roles also include self-run access. The default Manager role also includes running as other users. These permissions can be assigned through custom team roles. Scoped API keys use simulations:read, simulations:write, simulations:run, and simulations:run-as-others. API keys have no personal identity, so running simulations requires both run scopes. Live workflows also require the existing workflow execution permission. For more on team roles, see Permissions.