Bring your own agent

Use this when your agent is code: LangGraph, the OpenAI Agents SDK, CrewAI, or your own loop. Rehearsal does not run a model for such an agent. It runs the world, the simulated customer and the checks. Your code decides each action.

How it worksCopy Anchor Link

1. You create an agent profile that is marked as your own.
2. You start a run with it. The episodes start and wait for your agent.
3. For each waiting episode, your agent reads a turn: who it plays, what is new, and which tools it may call.
4. Your agent sends one action. Rehearsal carries it out in the world and returns the result.
5. This repeats until the episode ends. Then the checks run, as for any agent.

Set upCopy Anchor Link

rehearsal eval <wv_id> --profile <profile_id> --split dev --repeats 1

The loop in PythonCopy Anchor Link

import uuid
from rehearsal.sdk import Rehearsal

rh = Rehearsal()
run = rh.evaluations.create(WORLD_VERSION_ID, [AGENT_ID], splits=["dev"], repeats=1)
memory = {}  # (episode id, actor) -> everything that actor has seen

while rh.runs.get(run["run_id"])["status"] not in ("finished", "failed", "cancelled"):
    for episode in rh.runs.episodes(run["run_id"]):
        if episode["status"] != "running":
            continue
        observation = rh.external.observation(episode["id"])
        if not observation.get("awaiting_action"):
            continue
        turn = observation["turn"]["data"]
        history = memory.setdefault((episode["id"], turn["actor"]), [])
        history += turn["new"]

        tool, args = my_agent_decide(history, turn["tools"])  # your agent: exactly one tool call

        result = rh.external.act(episode["id"], turn["actor"], tool, args, idempotency_key=uuid.uuid4().hex)
        history.append({"role": "tool", "name": tool, "content": str(result)})

my_agent_decide is your code. It gets the conversation so far and the tools, and returns one tool name with its arguments.

• turn["tools"] uses the OpenAI function format: {"type": "function", "function": {"name", "description", "parameters"}}. Most agent frameworks accept it as their list of tools.
• turn["new"] has only what arrived since the actor's last action. Keep your own memory for each episode and actor, as the example does.
• A run can have several episodes waiting at once, one for each sandbox.

The repository has a complete example with a LangGraph ReAct agent: examples/agents/langgraph_react.py.

The loop over HTTPCopy Anchor Link

Every request carries Authorization: Bearer <your_key>.

1. Read the turnCopy Anchor Link

{
  "episode_status": "running",
  "awaiting_action": true,
  "turn": {
    "data": {
      "actor": "mira",
      "new": [{"role": "system", "content": "You are mira (maintainer). ..."}],
      "goal_version_observed": 1,
      "tools": [{"type": "function", "function": {"name": "create_comment", "description": "...", "parameters": {}}}]
    }
  }
}

The actor name and the tool are examples. new holds chat messages: the first turn has the role's instructions, and later turns have tool results, customer messages and messages from other actors. When awaiting_action is false, no turn is waiting: ask again in a second or two.

2. Send one actionCopy Anchor Link

{
  "actor": "mira",
  "tool": "create_comment",
  "args": {"index": 12, "body": "Thank you. The fix is in the next release."},
  "idempotency_key": "6f1c2a9e7b0d4c53"
}

The server answers 202 at once. The action is carried out in the world after that.

3. Get the resultCopy Anchor Link

{"status": "done", "result": {}}

While the action is in progress, the answer is {"status": "pending"}. Ask again in half a second.

What the idempotency key is forCopy Anchor Link

The key names one action. You choose it: 8 to 64 characters, different for each action. A random UUID is a good key.

• If your request is lost and you send the same action again with the same key, the server knows it and does not carry it out twice.
• You use the same key to get the result.
• The same key with a different action is refused with 409.

Errors from an actionCopy Anchor Link

Tools that every world offersCopy Anchor Link

Besides the application's own tools, a turn can offer these.

When the customer talks through the application itself, such as a chat channel or an issue comment, their messages arrive as notifications, and you answer with the application's own tools.

What a careful agent doesCopy Anchor Link

Your agent is tested on these. They are also good practice in production.

1. Read the customer's latest message before every write. Requests change during the work.
2. When a write returns "not carried out: a new message from the customer arrived", read the new message, then decide again.
3. After an UNKNOWN result, do not repeat the write blindly. Read the state first, or use retry_operation.
4. Find ids with read tools. Never guess an id.
5. Call complete only after a read shows that the work is done.
6. Do not contact the customer more than necessary.

A key for the agent onlyCopy Anchor Link

A key with "Build and evaluate" access can do everything on this page. To give agent code less, create a key with only the agent scope. It can read the waiting turn of an episode and send its action, and nothing else. Another program, with its own key, must then start the run and tell the agent the episode ids.

curl -X POST https://rehearsal.example.com/v1/api-keys \
  -H "Authorization: Bearer <a_full_access_key>" -H "Content-Type: application/json" \
  -d '{"name": "agent runner", "scopes": ["agent"]}'

Act as the agent from an AI assistantCopy Anchor Link

An AI assistant can play the agent through two MCP tools: waiting_turns returns the episodes that wait, and take_action sends one action. This is a good way to see how a careful agent behaves, or to check that a job is fair. See the MCP tool reference.

NextCopy Anchor Link