AgentRx: Diagnosing AI Agent Failures from Execution Trajectories
An AI agent run failed. Many steps went wrong along the way. At which step did it go wrong for good, and why?
key idea what the paper found what is lost, or a limit
chapter 1
The question in one picture
An LLM agent works in steps. At each step it writes a message or calls a tool, then reads what the tool returns. The full record of one run is its trajectory. §1 A failed run often holds more than one mistake, as in this run from the paper. C
One failed run, three kinds of failure C
A Magentic-One run tries to find data in a research paper. At step 3 the web agent cannot reach the documents. C
It fails again at steps 13 and 17. The lead agent sees the gap and hands the job to the file agent, so the run goes on. C
At step 33 the file agent reads a file path that does not exist, and the run never recovers. An earlier dataset marks step 3. The paper marks step 33. C
The run recovered from its early failures, but not from step 33. C The paper calls this the critical failure: the first failure in the run that no agent ever recovers from. §1 The step where it happens is the critical step. §2.1
So the paper asks: given a failed run, at which step did it fail for good, and why? By hand, this means reading the whole log. §1
The answer, in short
- A benchmark of 170 failed runs, each labeled by hand with its critical step and one of nine reasons. §2
- AgentRx: a tool that writes checks for each step, runs them, and gives the failed checks to a judge model. §3
- It finds the exact step more often than an earlier method, in all four parts of the benchmark. §4.2
In the run above, why is step 3 not the critical failure?
The run recovered from it. The lead agent handed the work to a different agent, and the run went on. The failure that could not be undone came later, at step 33. C
Why does it matter which failure you call the cause?
A developer fixes what the diagnosis points to. Fixing a failure the run already recovered from would not change the outcome. C
chapter 2
What you need first
Many failures, one cause
Some systems are one agent. Others are a team, where a lead agent, the orchestrator, hands out the work. §2 In a team, any agent can make the critical failure. §1 The paper calls any point where the agent stops moving toward a correct result a failure. §2.1 One run can have many failures. The paper wants only one of them: the one that decides the outcome. §2.1 So the first failure, the last failure and the critical failure can be three different steps. §4.1
A language model as judge
A judge is a language model asked to name the critical step and its reason. §3.4 An earlier method, Who&When, asks a model which agent failed and at which step. §4.2 AgentRx gives the judge more evidence. §3
A run fails at three steps. It recovers from the first one but not from the second. Which is the critical step?
The second. It is the earliest failure the run never recovers from. The third came after the point of no return. §2.1
What does AgentRx give the judge that a plain judge does not get?
A list of checks that failed, each with its step and the part of the log that shows it. §3.4
chapter 3
The benchmark
170 failed runs
The benchmark has 170 failed runs from 11 task settings, in four parts. §2 Τ-bench is a shop customer-service test. Flash is a team of agents that fixes cloud-service problems. Magentic-One is a team of agents for web and file tasks. §2 The fourth part, RelWork, takes runs from two earlier failure datasets, MAST and Who&When. §2 Labeling took 62.5 hours in all. §2.2
Where each part comes from, and how long the runs are
- Τ-bench: the authors ran 115 shop tasks with GPT-4o, one try each, and kept the 39 that failed. §2
- Flash: 42 runs with at least one agent failure. §2
- Magentic-One: 44 failed runs sampled at random from the Who&When dataset. §2
- RelWork: five runs from each of nine settings, seven from MAST and two from Who&When. That makes 45. §2
Median run lengths are 36 steps for τ-bench and 3 for Flash. §2.2 Magentic-One runs have a median of 33 steps. §2.2 Labeling took 20 to 24 minutes per run. §2.2
How the labels were made
Three people read the runs with no fixed list of labels at the start. They marked every failure, then picked the critical step, and grew categories from their notes. This is grounded theory. §2.1 Their mean Cohen's κ was 0.89. §2.2
How the list was frozen, and Figure 2
Only the first three parts shaped the list. The people merged notes into categories until new runs showed no new kind of failure, the point of theoretical saturation. §2.1 Then they froze the list and used it to label RelWork. So RelWork tests whether the list carries over to runs that did not shape it. §2.1 Agreement was measured on 30 runs, each labeled by all three people. §2.2
Figure 2 from the paper: every failure in every run Figure 2 open full size

Each gray line is one run of τ-bench, plotted against the step number. A star marks the critical step. Small squares mark other failures. Colors are failure categories.
Magentic-One: 295 failures in all. 68% of its runs have at least two failures, so the first failure is often not the critical one. Figure 2
Flash: short runs, so the stars sit close together. There are few steps to choose from.
Nine failure types
Each run's critical failure gets exactly one failure category. §2.2 Together the nine form the paper's failure taxonomy. §2.2
- Instruction/Plan Adherence Failure: the goal is right, but the agent skips a required step or adds one the plan did not include. §2.2 A
- Invention of New Information: the agent uses information that nothing in the run supports, or drops information that matters. §2.2
- Invalid Invocation: the agent calls a tool with inputs the tool cannot accept. §2.2
- Misinterpretation of Tool Output: the tool's output is fine, but the agent reasons about it wrongly. §2.2
- Intent–Plan Misalignment: the agent misreads the user's goal and makes a wrong plan. §2.2
- Under-specified User Intent: the agent lacks information it needs at that point. §2.2
- Intent Not Supported: the user asks for something no tool can do. §2.2
- Guardrails Triggered: a safety policy or an access block stops the agent. §2.2
- System Failure: a service the agent calls cannot be reached. §2.2
Types 1 and 5 look alike. The difference is whether the plan itself is right. §2.2 For each type, the taxonomy checklist lists a few yes-or-no questions that settle cases like this. §3 A
How common each type is
In Flash and Magentic-One, Misinterpretation of Tool Output is the most common critical failure, at 33.3% and 34.1% of runs. Table 1 In τ-bench, the most common is Under-specified User Intent, at 33.3%. Table 1 Guardrails Triggered appears mostly in Magentic-One, at 20.5% of its runs. Table 1 Invalid Invocation almost never is the critical failure: it appears only once, in RelWork. Table 1
The paper's list differs from the MAST list in purpose. MAST sorts failures by system design and how agents coordinate. This list labels one critical failure per run. §2.2
An agent understands the user's goal and makes a correct plan, then skips a confirmation step the shop rules require. Which type is it?
chapter 4
How the tool works
AgentRx takes three inputs: the tool schema, an optional domain policy, and the failed run. §3 It first converts each log into one shared format, an intermediate representation. §3
Writing check rules
Next, a language model writes constraints: rules that the steps should follow, in a form that can be checked. §3 Global constraints are written once from the tool schema and the domain policy. §3.1 Dynamic constraints are written at each step from the task and the run so far, such as what a tool returned. §3.1
A rule's guard says at which steps it applies. Its assertion is the test that passes or fails. §3.2 A programmatic constraint is checked by code. A semantic constraint is a plain-English rule that a language model checks. §3.2
Two constraints from the paper §3.2
From failed checks to an answer
A check that fails at a step is a violation. Each one goes into a validation log with its step and the part of the run that shows it. §3.3 The judge then reads the task, the run, the taxonomy checklist and the log. §3.4 It picks the first step whose failed checks are enough to explain why the run fails. §3.4
It may ignore a violation that the rest of the run shows does not matter. §3.4 Here is the paper's own example.
Figure 1 from the paper: how AgentRx finds a failure Figure 1 open full size

The inputs: the domain policy, the tool schema and the failed trajectory. §3
Step 6: a tool returns the T-shirt variants. Step 7: the agent says there are 11 available options. §1
A programmatic constraint for step 7 recounts the options in the tool's output. §1 G
The counts differ, so the check fails and goes into the validation log. §1
The judge reads the log and the taxonomy checklist and answers: Misinterpretation of Tool Output at step 7. Figure 1
Rule details, a tenth label, and two ways to write rules
Each rule names the failure types it targets, its trigger, a short hint, and either Python code or a model prompt. F The prompt for a semantic constraint tells the checking model to fail a step only on clear evidence. Missing or unclear evidence counts as a pass. F.2
The judge may also answer Inconclusive when none of the nine types fits. It must then name its own type. A §4.1
The default judge answers the step and the type in one call, the All-at-Once setup. The paper also tests Step-then-Category, which asks for the step first and the type second. §4.5
Why does each rule need a guard?
Most rules only make sense at some steps, such as when one tool is called. The guard skips the rule everywhere else, so it does not fail where it does not apply. §3.2
chapter 5
What they found
GPT-5 writes the rules and judges below, unless the page says otherwise. Results are means over three runs. §4.1
Finding the step: better than Who&When
An earlier method, W&W, found the exact critical step in 18.8% of τ-bench runs. How often do you think AgentRx did?
The part below opens after your guess.
The authors changed the W&W prompt to ask for the critical failure. §4.2 The measure is step accuracy: the share of runs where the predicted step is exactly right. §4.1
Step accuracy: W&W against AgentRx Table 3
AgentRx found the exact step more often on all four parts, by 75% on average in the paper's summary. §1 It also names a failure category, with a category accuracy from 46.2% to 65.9%. Table 3
The 75%, and looser scores
The paper does not show how it computed the 75%. A relative gain fits. On τ-bench, AgentRx scores about 2.3 times as high as W&W. Averaged over the four parts, the relative gains come to about 76%. That is our arithmetic; the paper gives only the 75%. Table 3
Table 3 also scores which agent failed. On Flash, W&W picked the right agent in 16.1% of runs, and AgentRx in 82.5%. Table 3
Table 7 counts a step as right when it is close. On τ-bench, within 5 steps, the judge without checks and AgentRx both score 68.4%. Table 7 So on τ-bench, much of the gain is in finding the exact step. Getting near it was already as good. Table 7
What the failed checks add
The judge-only baseline is the same GPT-5 judge with no failed checks. §4.3 On τ-bench, the failed checks raised step accuracy from 29.9% to 42.7%. Table 4 The taxonomy checklist helped most on Flash. §4.3
On τ-bench, which failure type do you think the judge never got right, with or without the failed checks?
The part below opens after your guess.
On τ-bench, the checks lifted System Failure from 0% to 66.7% correct. Table 5 Invention of New Information, Intent Not Supported and Under-specified User Intent stayed at 0% with and without the checks. Table 5 Instruction/Plan Adherence Failure fell a little. Without checks, the judge uses it as a catch-all. §4.3
Under-specified User Intent is the most common type in τ-bench. Table 1 Why does it stay at 0%? One guess (not from the paper): rules test what the agent did. A missing fact leaves no action for a rule to fail.
Writing rules step by step helps on long runs
Rules can be written in two ways. Step-by-step generation writes rules at each step from the run so far. One-shot generation writes all rules once from the whole run. §4.4 Step-by-step generation did better, most on long runs. §4.4 On Magentic-One, it cut the average step distance from 22.5 steps, the judge-only baseline's, to 12.4. §4.4
One-shot generation is cheaper. With GPT-5 prices, it costs USD 0.18 per run, against USD 0.05 for the judge-only baseline. §4.4
Other models, the full MAST set, and judge setups
Other models. With DeepSeek-V3.2 running the whole pipeline, the failed checks raised category accuracy on Flash and τ-bench by 5.6 to 8.7 pp. On Magentic-One and RelWork, results were mixed. Table 9 With o3 as the judge, the checks helped on τ-bench and Flash. Table 8
The full MAST set. MAST has no labels for this task, so the paper uses it as a stress test, not for accuracy. §4.2 AgentRx ran with no code changes on all 1,022 runs and gave each one a step, a type and evidence. §4.2 Porting it to a new agent system needs only a converter to the intermediate representation. §4.2
Figure 3 from the paper: failure types on 1,022 MAST runs Figure 3 open full size

In 7.6% of these runs, AgentRx gave a type with no match in the MAST list. §4.2 Failures in AppWorld came early in the run, and failures in Magentic-One came late. §4.2 Figure 4
Judge setups. Step-then-Category often raised category accuracy, but it could lower step accuracy: a wrong first step carries into the type. §4.5 So All-at-Once stays the default. B
Why did step-by-step generation help more on Magentic-One than on Flash?
Why could adding failed checks lower the score for one failure type?
Without checks, the judge puts many runs into Instruction/Plan Adherence Failure as a catch-all. With checks, some of those go to other types, so fewer land in it. §4.3
chapter 6
What this does not show
The paper states these limits itself. §7
- The list of types. Other agent settings may have failures it does not name. §7
- Misleading checks. Weak or false failed checks can mislead the judge. §7 In one Flash run, the cause was a System Failure, but most failed checks were about how the query was written. §4.7
- Late symptoms. In one Magentic-One run, a late safety block drew the judge to Guardrails Triggered. The cause was an earlier made-up download, Invention of New Information. §4.7
- Trust. A diagnosis can be trusted too much, and the list of types may share the blind spots of its benchmarks. Ethics
Points to check yourself (not from the paper). Exact step accuracy stays under 50% on τ-bench and Magentic-One. Each part has only 39 to 45 runs. The W&W results use a prompt the authors changed. Table 3
In the Magentic-One example, why did the judge pick the wrong type?
The failed check it relied on was real, but it came late in the run. It was a symptom of the earlier made-up download. The cause came earlier. §4.7
Why do the authors suggest finding a small set of strong checks?
Noisy or late failed checks can steer the judge away from the real cause. Fewer, better checks would separate real failures from false alarms. §7
chapter 7
Questions people ask
Short answers to common questions. Each one points back to the paper.
What is AgentRx?
It is a tool that reads a failed agent run. It names the step where the run went wrong for good, and the reason. §1 It writes check rules for the run, runs them, and gives the failed checks to a judge model. §3
What is a critical failure in an agent run?
It is the first failure in a run that no agent ever recovers from. §1 Earlier failures may be fixed later in the run, so the first failure is not always the cause. C
Why do AI agents fail?
In the paper's benchmark, common causes are misreading a tool's output, not following the plan, and missing information from the user. Table 1 The mix depends on the agent system. Table 1
How is AgentRx different from Who&When?
Who&When labels which agent failed and the step of the failure. §4.2 AgentRx looks for the first failure that cannot be undone, adds a failure category, and backs its answer with failed checks. Table 2 §4.2
Is the AgentRx benchmark public?
Yes. The paper releases the 170 labeled runs on Hugging Face and the code on GitHub. §1
keep it
This page has 12 check-yourself questions. The review page asks them again after 1, 3, 7 and 21 days, one at a time. You can also download them for Anki.
chapter 8
Words to know
Hover or tap any underlined word on the page to see its entry. These are the terms this page uses, in the order it introduces them.
- trajectory (run log)
- The full record of one agent run, step by step: messages, tool calls and what the tools returned."Each trajectory includes the full execution trace (messages, tool calls, tool outputs, and observable environment state)" (§1)
- Who&When
- An earlier dataset and method that marks which agent failed and at which step in multi-agent runs."Who&When is the closest benchmark for automated failure attribution, while MAST is the closest taxonomy-level analysis of multi-agent failures" (§4.2)
- orchestrator (lead agent)
- The agent that plans and hands out the work to the other agents in a team of agents."It uses an orchestrator to coordinate specialized agents for web browsing, file navigation, and code execution." (§2)
- critical failure (point of no return)
- The first mistake in a run that the agents never recover from."We define critical failure as the first unrecoverable failure by any agent in an execution trajectory." (§1)
- critical step (step of no return)
- The step number where the critical failure happens."We then identified the critical failure step: the earliest failure in the causal chain from which the agent does not recover." (§2.1)
- AgentRx
- The paper's tool: it reads a failed agent run and says at which step, and why, the run went wrong for good."we present AgentRx, an automated domain-agnostic diagnostic framework that pinpoints the critical failure step in a failed agent trajectory." (Abstract)
- judge (judge model)
- A language model given the run and other evidence, and asked to name the critical step and its failure type."The judge selects the first step whose violation evidence is sufficient to explain why the run fails with respect to I." (§3.4)
- RelWork
- The paper's name for 45 failed runs taken from two earlier failure datasets, MAST and Who&When."This yields 45 additional failed trajectories and brings the AgentRx benchmark to 170 trajectories in total." (§2)
- MAST
- An earlier study and dataset of failures in teams of agents, with its own list of failure types."MAST studies aggregate multi-agent failure patterns, while AgentRx diagnoses individual failed trajectories by localizing the first unrecoverable failure and grounding it in trace evidence." (§4.2)
- grounded theory (letting labels grow from the data)
- A way to build categories: read the data, write short notes, and merge the notes into categories, instead of starting from a fixed list."we manually annotate trajectories using a grounded-theory open coding process (Glaser and Strauss, 2017), allowing failure categories to emerge from the traces rather than imposing a predefined label set." (§2.1)
- Cohen's κ (agreement score)
- A score for how often two people give the same label, after removing the agreement you would get by chance. 1 is perfect agreement."Because Cohen's κ is defined for two annotators, we compute it pairwise across annotator pairs and report the mean." (§2.2)
- theoretical saturation (the point where nothing new turns up)
- The point where reading more runs no longer turns up a new kind of failure."We iterated until reaching theoretical saturation, where new trajectories no longer introduced new failure phenomena." (§2.1)
- failure category (failure type)
- A label that says why the run failed at its critical step."Each trajectory includes the full execution trace (messages, tool calls, tool outputs, and observable environment state) and is annotated with the critical step and a failure category capturing why the run ultimately fails." (§1)
- failure taxonomy (list of failure types)
- The paper's fixed list of nine failure types, each with a definition."Our grounded-theory-based annotation process also yields a cross-domain failure taxonomy for root-cause attribution." (§1)
- Instruction/Plan Adherence Failure (not following the plan)
- The goal is right, but the agent skips a required step or adds one that was not in the plan."The agent fails to follow required directions or the agreed plan by ignoring directives. This covers both under-execution and over-execution (unplanned actions) that violate the plan, domain policy, or orchestrator constraints." (§2.2)
- Invention of New Information (made-up information)
- The agent uses a fact, value or file that nothing in the run supports, or drops one that matters."The agent introduces, removes, or alters information that is not grounded in any available input, context, or tool output." (§2.2)
- Invalid Invocation (broken tool call)
- The agent calls a tool with inputs the tool cannot accept."The agent issues an invalid tool call, e.g., malformed inputs, missing arguments, or values that fail schema validation." (§2.2)
- Misinterpretation of Tool Output (misreading a tool's output)
- The tool returned the right thing, but the agent read it wrong."The agent incorrectly reasons about its own or another agent's tool output, leading to incorrect assumptions or actions." (§2.2)
- Intent–Plan Misalignment (a wrong plan for the goal)
- The agent misunderstands what the user wants, so its plan is wrong from the start."The agent misinterprets the user's goal or constraints and produces an incorrect plan." (§2.2)
- Under-specified User Intent (missing information from the user)
- The agent could not finish because it did not have all the information it needed at that point."The agent was unable to complete the task due to lack of complete information at that point in the trajectory." (§2.2)
- Intent Not Supported (no tool for the request)
- The user asks for something that no available tool can do."The agent is asked to perform an action for which there is no tool." (§2.2)
- Guardrails Triggered (blocked by a safety rule)
- A safety policy, a login wall or a similar block stops the agent."The agent's execution is blocked by safety policies or external access restrictions." (§2.2)
- System Failure (a system or connection error)
- The agent did nothing wrong: a service it called could not be reached."The agent faces a system connectivity issue while calling a particular tool like an API endpoint not being reachable." (§2.2)
- taxonomy checklist (yes-or-no question list)
- For each failure type, a few yes-or-no questions the judge answers before it picks that type."We manually compile our failure taxonomy into a semantic checklist K: for each category y in the taxonomy, K(y) specifies a small set of targeted yes/no questions (and brief decision criteria) that operationalize the category definition." (§3)
- tool schema (tool descriptions)
- The list of tools an agent can call, with the inputs each one takes and the outputs it returns."AgentRx takes as input (i) a toolset T with input–output schemas, (ii) an optional domain policy Π, and (iii) a failed trajectory T=⟨s1,...,sn⟩." (§3)
- domain policy (rule book)
- Written rules for the task area, such as "confirm with the user before you cancel an order"."Domain policies are optional: τ-bench provides a policy document, while Flash and Magentic-One provide none, so their constraints derive from tool schemas and the observed trajectory alone." (§4.1)
- intermediate representation (shared log format)
- One common format that logs from different agent systems are first converted into."Since log formats differ across domains, we first normalize each trajectory into a common intermediate representation (IR)." (§3)
- constraint (check rule)
- A rule that a step of the run should follow, written so a program or a model can check it."AgentRx generates executable constraints from the tool schema, domain policy, and trajectory prefixes, then evaluates them step-by-step to produce an auditable validation log V." (§3)
- global constraint (task-wide rule)
- A check rule written once, from the tool descriptions and the rule book, that applies to every run in the task area."Intuitively, global constraints encode domain rules, while dynamic constraints encode trajectory-specific rules and prior observations (e.g., constraints that must hold given earlier tool outputs)." (§3.1)
- dynamic constraint (run-specific rule)
- A check rule written at a given step, from what this run has shown so far, such as what a tool just returned."Intuitively, global constraints encode domain rules, while dynamic constraints encode trajectory-specific rules and prior observations (e.g., constraints that must hold given earlier tool outputs)." (§3.1)
- guard (when-to-check condition)
- The part of a check rule that says at which steps the rule applies."Guards capture structural conditions such as whether step k calls a particular tool or invokes a particular agent." (§3.2)
- assertion (test)
- The part of a check rule that returns pass or fail."A constraint C generated by AgentRx comprises (i) a guard G_C(T≤k, s_k) ∈ {0,1} that determines when the constraint applies and (ii) an assertion Φ_C(T≤k, s_k) ∈ {SAT, VIOL} that returns a binary verdict v when evaluated." (§3.2)
- programmatic constraint (code check)
- A check rule written as a small program, such as "the count the agent states equals the count in the tool's output"."Constraints are checked either programmatically over structured fields (e.g., schema validity, equality, membership), or semantically as natural-language predicates evaluated using an LLM-based checker." (§3.2)
- semantic constraint (model-read check)
- A check rule written in plain English that a language model reads and judges pass or fail."Constraints are checked either programmatically over structured fields (e.g., schema validity, equality, membership), or semantically as natural-language predicates evaluated using an LLM-based checker." (§3.2)
- violation (failed check)
- A check rule that applied at a step and failed there."We record each violated constraint and its supporting evidence in a step-indexed validation log" (§3.3)
- validation log (list of failed checks)
- A list of every failed check, with its step number and the part of the run that shows the failure."We record each violated constraint and its supporting evidence in a step-indexed validation log" (§3.3)
- Inconclusive (no clear type)
- A tenth label the judge may use when none of the nine types fits; it must then name its own type."None of 1--9 clearly apply; the judge must provide a custom category label." (App. A)
- All-at-Once (both answers in one call)
- The judge names the step and the failure type in one answer."We compare two LLM-as-judge protocols: All-at-Once, which predicts the critical step and category in one call, and Step-then-Category, which first predicts the step and then categorizes that step." (§4.5)
- Step-then-Category (step first, type second)
- The judge first names the step, then, in a second call, the failure type of that step."We compare two LLM-as-judge protocols: All-at-Once, which predicts the critical step and category in one call, and Step-then-Category, which first predicts the step and then categorizes that step." (§4.5)
- step accuracy (share of exact steps)
- The share of runs where the predicted step is exactly the step the people marked."We evaluate step localization using exact critical-step accuracy (the fraction of trajectories where the predicted step exactly matches the annotated step)" (§4.1)
- category accuracy (share of right types)
- The share of runs where the predicted failure type matches the type people gave the critical failure."We evaluate category prediction using critical, any-failure, earliest, and terminal accuracy, comparing the predicted category against the annotated critical failure, any annotated failure, the first observed failure, and the terminal failure, respectively." (§4.1)
- judge-only baseline (judge with no checks)
- The same judge model, given the run and the failure types, but no list of failed checks."Table 4 compares AgentRx against a judge-only baseline that sees the trajectory and taxonomy prompt, but no AgentRx-generated validation evidence." (§4.3)
- step-by-step generation (writing rules at each step)
- Writing new check rules at every step, from only the part of the run up to that step."We evaluate two constraint-generation strategies: one-shot, which generates constraints once from the full trajectory, and step-by-step, which generates constraints from each prefix T≤k." (§4.4)
- one-shot generation (writing rules once)
- Writing all the check rules in one pass, from the whole run."We evaluate two constraint-generation strategies: one-shot, which generates constraints once from the full trajectory, and step-by-step, which generates constraints from each prefix T≤k." (§4.4)
- average step distance (average steps off)
- On average, how many steps the predicted step is from the marked one. Lower is better."and average step distance, which measures how far predictions are from the annotated step." (§4.1)
- LLM (language model)
- A large language model: a model trained on a lot of text that answers in text."Large Language Model (LLM)-based agents are increasingly acting autonomously without a human in the loop." (§1)
- WebSurfer
- The web-browsing agent in Magentic-One."It uses an orchestrator to coordinate specialized agents for web browsing, file navigation, and code execution." (§2)
- FileSurfer
- The file-reading agent in Magentic-One."It uses an orchestrator to coordinate specialized agents for web browsing, file navigation, and code execution." (§2)
- Magentic-One
- A team of agents for open web and file tasks, led by an orchestrator."Magentic-One (Fourney et al., 2024) is a generalist multi-agent system for open-ended web and file-based tasks." (§2)
- τ-bench
- A test set of shop customer-service tasks: one agent, a simulated customer, store tools and a rule book."τ-bench (Yao et al., 2024) is a tool-calling benchmark for retail tasks, with an LLM-simulated user and a single agent equipped with domain-specific APIs and policy guidelines." (§2)
- Flash
- A team of agents that follows a troubleshooting guide for repeat problems in cloud services."Flash (Zhang et al., 2024) is a workflow automation agent for diagnosing recurring incidents in production cloud services." (§2)
- GPT-4o
- An OpenAI language model; it ran the agent in the τ-bench runs."We run 115 τ-bench retail tasks using gpt-4o with one trial and analyze the 39 failed trajectories." (§2)
- GPT-5
- An OpenAI language model; the paper's default model for writing the check rules and for the judge."We use gpt-5 as the default model." (§4.1)
- DeepSeek-V3.2
- An open-weight language model from DeepSeek, tried in place of GPT-5 for the whole pipeline."Failure-category accuracy with DeepSeek-V3.2 as the model for both constraint generation and judging." (App. A)
- o3
- An OpenAI reasoning model, tried as a second judge model."o3 judge ablation (mean ± std, n=3; configurations as in Table 4)." (App. A, Table 8)