← library12345678 words font

AgentRx: Diagnosing AI Agent Failures from Execution Trajectories

Shraddha Barke, Arnav Goyal, Alind Khare, Avaljot Singh, Suman Nath, Chetan Bansal · arXiv preprint 2602.02475, v2 (arXiv comment: accepted to EMNLP Findings 2026) 2026

about 9.5 minthe paper (PDF)reviewnotes written 2026-11-16

An AI agent run failed. Many steps went wrong along the way. At which step did it go wrong for good, and why?

key idea what the paper found what is lost, or a limit

chapter 1

The question in one picture

An LLM agent works in steps. At each step it writes a message or calls a tool, then reads what the tool returns. The full record of one run is its trajectory. §1 A failed run often holds more than one mistake, as in this run from the paper. C

One failed run, three kinds of failure C

step 3, WebSurfercannot reach the documents it needsWho&When marks this step
steps 13 and 17, WebSurferdoes not download and search the PDF it was told to
the orchestratorswitches the job to FileSurferthe run recovers
step 33, FileSurfersaves the file at one path, then reads another path that does not existthe paper marks this step

A Magentic-One run tries to find data in a research paper. At step 3 the web agent cannot reach the documents. C

It fails again at steps 13 and 17. The lead agent sees the gap and hands the job to the file agent, so the run goes on. C

At step 33 the file agent reads a file path that does not exist, and the run never recovers. An earlier dataset marks step 3. The paper marks step 33. C

The run recovered from its early failures, but not from step 33. C The paper calls this the critical failure: the first failure in the run that no agent ever recovers from. §1 The step where it happens is the critical step. §2.1

So the paper asks: given a failed run, at which step did it fail for good, and why? By hand, this means reading the whole log. §1

The answer, in short

check yourself

In the run above, why is step 3 not the critical failure?

Why does it matter which failure you call the cause?

chapter 2

What you need first

Many failures, one cause

Some systems are one agent. Others are a team, where a lead agent, the orchestrator, hands out the work. §2 In a team, any agent can make the critical failure. §1 The paper calls any point where the agent stops moving toward a correct result a failure. §2.1 One run can have many failures. The paper wants only one of them: the one that decides the outcome. §2.1 So the first failure, the last failure and the critical failure can be three different steps. §4.1

A language model as judge

A judge is a language model asked to name the critical step and its reason. §3.4 An earlier method, Who&When, asks a model which agent failed and at which step. §4.2 AgentRx gives the judge more evidence. §3

check yourself

A run fails at three steps. It recovers from the first one but not from the second. Which is the critical step?

What does AgentRx give the judge that a plain judge does not get?

chapter 3

The benchmark

170 failed runs

The benchmark has 170 failed runs from 11 task settings, in four parts. §2 Τ-bench is a shop customer-service test. Flash is a team of agents that fixes cloud-service problems. Magentic-One is a team of agents for web and file tasks. §2 The fourth part, RelWork, takes runs from two earlier failure datasets, MAST and Who&When. §2 Labeling took 62.5 hours in all. §2.2

Where each part comes from, and how long the runs are
  • Τ-bench: the authors ran 115 shop tasks with GPT-4o, one try each, and kept the 39 that failed. §2
  • Flash: 42 runs with at least one agent failure. §2
  • Magentic-One: 44 failed runs sampled at random from the Who&When dataset. §2
  • RelWork: five runs from each of nine settings, seven from MAST and two from Who&When. That makes 45. §2

Median run lengths are 36 steps for τ-bench and 3 for Flash. §2.2 Magentic-One runs have a median of 33 steps. §2.2 Labeling took 20 to 24 minutes per run. §2.2

How the labels were made

Three people read the runs with no fixed list of labels at the start. They marked every failure, then picked the critical step, and grew categories from their notes. This is grounded theory. §2.1 Their mean Cohen's κ was 0.89. §2.2

How the list was frozen, and Figure 2

Only the first three parts shaped the list. The people merged notes into categories until new runs showed no new kind of failure, the point of theoretical saturation. §2.1 Then they froze the list and used it to label RelWork. So RelWork tests whether the list carries over to runs that did not shape it. §2.1 Agreement was measured on 30 runs, each labeled by all three people. §2.2

Figure 2 from the paper: every failure in every run Figure 2 open full size

Figure 2 from the paper: every failure in every run

Each gray line is one run of τ-bench, plotted against the step number. A star marks the critical step. Small squares mark other failures. Colors are failure categories.

Magentic-One: 295 failures in all. 68% of its runs have at least two failures, so the first failure is often not the critical one. Figure 2

Flash: short runs, so the stars sit close together. There are few steps to choose from.

Nine failure types

Each run's critical failure gets exactly one failure category. §2.2 Together the nine form the paper's failure taxonomy. §2.2

  1. Instruction/Plan Adherence Failure: the goal is right, but the agent skips a required step or adds one the plan did not include. §2.2 A
  2. Invention of New Information: the agent uses information that nothing in the run supports, or drops information that matters. §2.2
  3. Invalid Invocation: the agent calls a tool with inputs the tool cannot accept. §2.2
  4. Misinterpretation of Tool Output: the tool's output is fine, but the agent reasons about it wrongly. §2.2
  5. Intent–Plan Misalignment: the agent misreads the user's goal and makes a wrong plan. §2.2
  6. Under-specified User Intent: the agent lacks information it needs at that point. §2.2
  7. Intent Not Supported: the user asks for something no tool can do. §2.2
  8. Guardrails Triggered: a safety policy or an access block stops the agent. §2.2
  9. System Failure: a service the agent calls cannot be reached. §2.2

Types 1 and 5 look alike. The difference is whether the plan itself is right. §2.2 For each type, the taxonomy checklist lists a few yes-or-no questions that settle cases like this. §3 A

How common each type is

In Flash and Magentic-One, Misinterpretation of Tool Output is the most common critical failure, at 33.3% and 34.1% of runs. Table 1 In τ-bench, the most common is Under-specified User Intent, at 33.3%. Table 1 Guardrails Triggered appears mostly in Magentic-One, at 20.5% of its runs. Table 1 Invalid Invocation almost never is the critical failure: it appears only once, in RelWork. Table 1

The paper's list differs from the MAST list in purpose. MAST sorts failures by system design and how agents coordinate. This list labels one critical failure per run. §2.2

check yourself

An agent understands the user's goal and makes a correct plan, then skips a confirmation step the shop rules require. Which type is it?

An agent asks a shop tool to cancel an order but leaves out the order number, and the tool rejects the call. Which type is it?

chapter 4

How the tool works

AgentRx takes three inputs: the tool schema, an optional domain policy, and the failed run. §3 It first converts each log into one shared format, an intermediate representation. §3

Writing check rules

Next, a language model writes constraints: rules that the steps should follow, in a form that can be checked. §3 Global constraints are written once from the tool schema and the domain policy. §3.1 Dynamic constraints are written at each step from the task and the run so far, such as what a tool returned. §3.1

A rule's guard says at which steps it applies. Its assertion is the test that passes or fails. §3.2 A programmatic constraint is checked by code. A semantic constraint is a plain-English rule that a language model checks. §3.2

Two constraints from the paper §3.2

guard: when the rule applies
assertion: what must be true
programmatic constraintrun as code
step 7, when the agent's message mentions T-shirts
the number of options the agent states equals the number of available variants in the latest product lookup
semantic constraintread by a language model
the agent calls a tool that changes something: cancel, return, exchange, modify an order or an address
earlier, the agent described the action and the order, and the user clearly said yes
If the guard does not apply, the rule is skipped. If the test fails, the step, the rule and the evidence are written down.

Two parts per rule. §3.2

A small Python function over the log's fields. G

A plain-English rule from the shop's domain policy, judged by a language model. F.2

Both kinds return a verdict and its evidence. §3.2

From failed checks to an answer

A check that fails at a step is a violation. Each one goes into a validation log with its step and the part of the run that shows it. §3.3 The judge then reads the task, the run, the taxonomy checklist and the log. §3.4 It picks the first step whose failed checks are enough to explain why the run fails. §3.4

It may ignore a violation that the rest of the run shows does not matter. §3.4 Here is the paper's own example.

Figure 1 from the paper: how AgentRx finds a failure Figure 1 open full size

Figure 1 from the paper: how  AgentRx  finds a failure

The inputs: the domain policy, the tool schema and the failed trajectory. §3

Step 6: a tool returns the T-shirt variants. Step 7: the agent says there are 11 available options. §1

A programmatic constraint for step 7 recounts the options in the tool's output. §1 G

The counts differ, so the check fails and goes into the validation log. §1

The judge reads the log and the taxonomy checklist and answers: Misinterpretation of Tool Output at step 7. Figure 1

Rule details, a tenth label, and two ways to write rules

Each rule names the failure types it targets, its trigger, a short hint, and either Python code or a model prompt. F The prompt for a semantic constraint tells the checking model to fail a step only on clear evidence. Missing or unclear evidence counts as a pass. F.2

The judge may also answer Inconclusive when none of the nine types fits. It must then name its own type. A §4.1

The default judge answers the step and the type in one call, the All-at-Once setup. The paper also tests Step-then-Category, which asks for the step first and the type second. §4.5

check yourself

Why does each rule need a guard?

Why may the judge overrule a violation?

chapter 5

What they found

GPT-5 writes the rules and judges below, unless the page says otherwise. Results are means over three runs. §4.1

Finding the step: better than Who&When

guess first

An earlier method, W&W, found the exact critical step in 18.8% of τ-bench runs. How often do you think AgentRx did?

The part below opens after your guess.

The authors changed the W&W prompt to ask for the critical failure. §4.2 The measure is step accuracy: the share of runs where the predicted step is exactly right. §4.1

Step accuracy: W&W against AgentRx Table 3

part of the benchmark
W&W, prompt changed
AgentRx
τ-bench
18.8%
42.7%
Magentic-One
22.7%
36.4%
Flash
64.3%
82.5%
RelWork
32.6%
61.5%

On τ-bench, AgentRx finds the exact critical step in 42.7% of runs, against 18.8%. Table 3

It is higher on all four parts. The long Magentic-One runs have the lowest score. Table 3

AgentRx found the exact step more often on all four parts, by 75% on average in the paper's summary. §1 It also names a failure category, with a category accuracy from 46.2% to 65.9%. Table 3

The 75%, and looser scores

The paper does not show how it computed the 75%. A relative gain fits. On τ-bench, AgentRx scores about 2.3 times as high as W&W. Averaged over the four parts, the relative gains come to about 76%. That is our arithmetic; the paper gives only the 75%. Table 3

Table 3 also scores which agent failed. On Flash, W&W picked the right agent in 16.1% of runs, and AgentRx in 82.5%. Table 3

Table 7 counts a step as right when it is close. On τ-bench, within 5 steps, the judge without checks and AgentRx both score 68.4%. Table 7 So on τ-bench, much of the gain is in finding the exact step. Getting near it was already as good. Table 7

What the failed checks add

The judge-only baseline is the same GPT-5 judge with no failed checks. §4.3 On τ-bench, the failed checks raised step accuracy from 29.9% to 42.7%. Table 4 The taxonomy checklist helped most on Flash. §4.3

guess first

On τ-bench, which failure type do you think the judge never got right, with or without the failed checks?

The part below opens after your guess.

On τ-bench, the checks lifted System Failure from 0% to 66.7% correct. Table 5 Invention of New Information, Intent Not Supported and Under-specified User Intent stayed at 0% with and without the checks. Table 5 Instruction/Plan Adherence Failure fell a little. Without checks, the judge uses it as a catch-all. §4.3

Under-specified User Intent is the most common type in τ-bench. Table 1 Why does it stay at 0%? One guess (not from the paper): rules test what the agent did. A missing fact leaves no action for a rule to fail.

Writing rules step by step helps on long runs

Rules can be written in two ways. Step-by-step generation writes rules at each step from the run so far. One-shot generation writes all rules once from the whole run. §4.4 Step-by-step generation did better, most on long runs. §4.4 On Magentic-One, it cut the average step distance from 22.5 steps, the judge-only baseline's, to 12.4. §4.4

One-shot generation is cheaper. With GPT-5 prices, it costs USD 0.18 per run, against USD 0.05 for the judge-only baseline. §4.4

Other models, the full MAST set, and judge setups

Other models. With DeepSeek-V3.2 running the whole pipeline, the failed checks raised category accuracy on Flash and τ-bench by 5.6 to 8.7 pp. On Magentic-One and RelWork, results were mixed. Table 9 With o3 as the judge, the checks helped on τ-bench and Flash. Table 8

The full MAST set. MAST has no labels for this task, so the paper uses it as a stress test, not for accuracy. §4.2 AgentRx ran with no code changes on all 1,022 runs and gave each one a step, a type and evidence. §4.2 Porting it to a new agent system needs only a converter to the intermediate representation. §4.2

Figure 3 from the paper: failure types on 1,022 MAST runs Figure 3 open full size

Figure 3 from the paper: failure types on 1,022  MAST  runs

Each row is one agent framework, as a percent of its runs. Plan Adherence is the largest type overall. §4.2

Some types are large in only one framework: Invalid Invocation in MetaGPT, Misinterpretation of Tool Output in AppWorld, System Failure in OpenManus. §4.2

In 7.6% of these runs, AgentRx gave a type with no match in the MAST list. §4.2 Failures in AppWorld came early in the run, and failures in Magentic-One came late. §4.2 Figure 4

Judge setups. Step-then-Category often raised category accuracy, but it could lower step accuracy: a wrong first step carries into the type. §4.5 So All-at-Once stays the default. B

check yourself

Why did step-by-step generation help more on Magentic-One than on Flash?

Why could adding failed checks lower the score for one failure type?

chapter 6

What this does not show

The paper states these limits itself. §7

Points to check yourself (not from the paper). Exact step accuracy stays under 50% on τ-bench and Magentic-One. Each part has only 39 to 45 runs. The W&W results use a prompt the authors changed. Table 3

check yourself

In the Magentic-One example, why did the judge pick the wrong type?

Why do the authors suggest finding a small set of strong checks?

chapter 7

Questions people ask

Short answers to common questions. Each one points back to the paper.

What is AgentRx?

It is a tool that reads a failed agent run. It names the step where the run went wrong for good, and the reason. §1 It writes check rules for the run, runs them, and gives the failed checks to a judge model. §3

What is a critical failure in an agent run?

It is the first failure in a run that no agent ever recovers from. §1 Earlier failures may be fixed later in the run, so the first failure is not always the cause. C

Why do AI agents fail?

In the paper's benchmark, common causes are misreading a tool's output, not following the plan, and missing information from the user. Table 1 The mix depends on the agent system. Table 1

How is AgentRx different from Who&When?

Who&When labels which agent failed and the step of the failure. §4.2 AgentRx looks for the first failure that cannot be undone, adds a failure category, and backs its answer with failed checks. Table 2 §4.2

Is the AgentRx benchmark public?

Yes. The paper releases the 170 labeled runs on Hugging Face and the code on GitHub. §1

keep it

This page has 12 check-yourself questions. The review page asks them again after 1, 3, 7 and 21 days, one at a time. You can also download them for Anki.

add these 12 questions to my review

chapter 8

Words to know

Hover or tap any underlined word on the page to see its entry. These are the terms this page uses, in the order it introduces them.

trajectory (run log)
The full record of one agent run, step by step: messages, tool calls and what the tools returned.
"Each trajectory includes the full execution trace (messages, tool calls, tool outputs, and observable environment state)" (§1)
Who&When
An earlier dataset and method that marks which agent failed and at which step in multi-agent runs.
"Who&When is the closest benchmark for automated failure attribution, while MAST is the closest taxonomy-level analysis of multi-agent failures" (§4.2)
orchestrator (lead agent)
The agent that plans and hands out the work to the other agents in a team of agents.
"It uses an orchestrator to coordinate specialized agents for web browsing, file navigation, and code execution." (§2)
critical failure (point of no return)
The first mistake in a run that the agents never recover from.
"We define critical failure as the first unrecoverable failure by any agent in an execution trajectory." (§1)
critical step (step of no return)
The step number where the critical failure happens.
"We then identified the critical failure step: the earliest failure in the causal chain from which the agent does not recover." (§2.1)
AgentRx
The paper's tool: it reads a failed agent run and says at which step, and why, the run went wrong for good.
"we present AgentRx, an automated domain-agnostic diagnostic framework that pinpoints the critical failure step in a failed agent trajectory." (Abstract)
judge (judge model)
A language model given the run and other evidence, and asked to name the critical step and its failure type.
"The judge selects the first step whose violation evidence is sufficient to explain why the run fails with respect to I." (§3.4)
RelWork
The paper's name for 45 failed runs taken from two earlier failure datasets, MAST and Who&When.
"This yields 45 additional failed trajectories and brings the AgentRx benchmark to 170 trajectories in total." (§2)
MAST
An earlier study and dataset of failures in teams of agents, with its own list of failure types.
"MAST studies aggregate multi-agent failure patterns, while AgentRx diagnoses individual failed trajectories by localizing the first unrecoverable failure and grounding it in trace evidence." (§4.2)
grounded theory (letting labels grow from the data)
A way to build categories: read the data, write short notes, and merge the notes into categories, instead of starting from a fixed list.
"we manually annotate trajectories using a grounded-theory open coding process (Glaser and Strauss, 2017), allowing failure categories to emerge from the traces rather than imposing a predefined label set." (§2.1)
Cohen's κ (agreement score)
A score for how often two people give the same label, after removing the agreement you would get by chance. 1 is perfect agreement.
"Because Cohen's κ is defined for two annotators, we compute it pairwise across annotator pairs and report the mean." (§2.2)
theoretical saturation (the point where nothing new turns up)
The point where reading more runs no longer turns up a new kind of failure.
"We iterated until reaching theoretical saturation, where new trajectories no longer introduced new failure phenomena." (§2.1)
failure category (failure type)
A label that says why the run failed at its critical step.
"Each trajectory includes the full execution trace (messages, tool calls, tool outputs, and observable environment state) and is annotated with the critical step and a failure category capturing why the run ultimately fails." (§1)
failure taxonomy (list of failure types)
The paper's fixed list of nine failure types, each with a definition.
"Our grounded-theory-based annotation process also yields a cross-domain failure taxonomy for root-cause attribution." (§1)
Instruction/Plan Adherence Failure (not following the plan)
The goal is right, but the agent skips a required step or adds one that was not in the plan.
"The agent fails to follow required directions or the agreed plan by ignoring directives. This covers both under-execution and over-execution (unplanned actions) that violate the plan, domain policy, or orchestrator constraints." (§2.2)
Invention of New Information (made-up information)
The agent uses a fact, value or file that nothing in the run supports, or drops one that matters.
"The agent introduces, removes, or alters information that is not grounded in any available input, context, or tool output." (§2.2)
Invalid Invocation (broken tool call)
The agent calls a tool with inputs the tool cannot accept.
"The agent issues an invalid tool call, e.g., malformed inputs, missing arguments, or values that fail schema validation." (§2.2)
Misinterpretation of Tool Output (misreading a tool's output)
The tool returned the right thing, but the agent read it wrong.
"The agent incorrectly reasons about its own or another agent's tool output, leading to incorrect assumptions or actions." (§2.2)
Intent–Plan Misalignment (a wrong plan for the goal)
The agent misunderstands what the user wants, so its plan is wrong from the start.
"The agent misinterprets the user's goal or constraints and produces an incorrect plan." (§2.2)
Under-specified User Intent (missing information from the user)
The agent could not finish because it did not have all the information it needed at that point.
"The agent was unable to complete the task due to lack of complete information at that point in the trajectory." (§2.2)
Intent Not Supported (no tool for the request)
The user asks for something that no available tool can do.
"The agent is asked to perform an action for which there is no tool." (§2.2)
Guardrails Triggered (blocked by a safety rule)
A safety policy, a login wall or a similar block stops the agent.
"The agent's execution is blocked by safety policies or external access restrictions." (§2.2)
System Failure (a system or connection error)
The agent did nothing wrong: a service it called could not be reached.
"The agent faces a system connectivity issue while calling a particular tool like an API endpoint not being reachable." (§2.2)
taxonomy checklist (yes-or-no question list)
For each failure type, a few yes-or-no questions the judge answers before it picks that type.
"We manually compile our failure taxonomy into a semantic checklist K: for each category y in the taxonomy, K(y) specifies a small set of targeted yes/no questions (and brief decision criteria) that operationalize the category definition." (§3)
tool schema (tool descriptions)
The list of tools an agent can call, with the inputs each one takes and the outputs it returns.
"AgentRx takes as input (i) a toolset T with input–output schemas, (ii) an optional domain policy Π, and (iii) a failed trajectory T=⟨s1,...,sn⟩." (§3)
domain policy (rule book)
Written rules for the task area, such as "confirm with the user before you cancel an order".
"Domain policies are optional: τ-bench provides a policy document, while Flash and Magentic-One provide none, so their constraints derive from tool schemas and the observed trajectory alone." (§4.1)
intermediate representation (shared log format)
One common format that logs from different agent systems are first converted into.
"Since log formats differ across domains, we first normalize each trajectory into a common intermediate representation (IR)." (§3)
constraint (check rule)
A rule that a step of the run should follow, written so a program or a model can check it.
"AgentRx generates executable constraints from the tool schema, domain policy, and trajectory prefixes, then evaluates them step-by-step to produce an auditable validation log V." (§3)
global constraint (task-wide rule)
A check rule written once, from the tool descriptions and the rule book, that applies to every run in the task area.
"Intuitively, global constraints encode domain rules, while dynamic constraints encode trajectory-specific rules and prior observations (e.g., constraints that must hold given earlier tool outputs)." (§3.1)
dynamic constraint (run-specific rule)
A check rule written at a given step, from what this run has shown so far, such as what a tool just returned.
"Intuitively, global constraints encode domain rules, while dynamic constraints encode trajectory-specific rules and prior observations (e.g., constraints that must hold given earlier tool outputs)." (§3.1)
guard (when-to-check condition)
The part of a check rule that says at which steps the rule applies.
"Guards capture structural conditions such as whether step k calls a particular tool or invokes a particular agent." (§3.2)
assertion (test)
The part of a check rule that returns pass or fail.
"A constraint C generated by AgentRx comprises (i) a guard G_C(T≤k, s_k) ∈ {0,1} that determines when the constraint applies and (ii) an assertion Φ_C(T≤k, s_k) ∈ {SAT, VIOL} that returns a binary verdict v when evaluated." (§3.2)
programmatic constraint (code check)
A check rule written as a small program, such as "the count the agent states equals the count in the tool's output".
"Constraints are checked either programmatically over structured fields (e.g., schema validity, equality, membership), or semantically as natural-language predicates evaluated using an LLM-based checker." (§3.2)
semantic constraint (model-read check)
A check rule written in plain English that a language model reads and judges pass or fail.
"Constraints are checked either programmatically over structured fields (e.g., schema validity, equality, membership), or semantically as natural-language predicates evaluated using an LLM-based checker." (§3.2)
violation (failed check)
A check rule that applied at a step and failed there.
"We record each violated constraint and its supporting evidence in a step-indexed validation log" (§3.3)
validation log (list of failed checks)
A list of every failed check, with its step number and the part of the run that shows the failure.
"We record each violated constraint and its supporting evidence in a step-indexed validation log" (§3.3)
Inconclusive (no clear type)
A tenth label the judge may use when none of the nine types fits; it must then name its own type.
"None of 1--9 clearly apply; the judge must provide a custom category label." (App. A)
All-at-Once (both answers in one call)
The judge names the step and the failure type in one answer.
"We compare two LLM-as-judge protocols: All-at-Once, which predicts the critical step and category in one call, and Step-then-Category, which first predicts the step and then categorizes that step." (§4.5)
Step-then-Category (step first, type second)
The judge first names the step, then, in a second call, the failure type of that step.
"We compare two LLM-as-judge protocols: All-at-Once, which predicts the critical step and category in one call, and Step-then-Category, which first predicts the step and then categorizes that step." (§4.5)
step accuracy (share of exact steps)
The share of runs where the predicted step is exactly the step the people marked.
"We evaluate step localization using exact critical-step accuracy (the fraction of trajectories where the predicted step exactly matches the annotated step)" (§4.1)
category accuracy (share of right types)
The share of runs where the predicted failure type matches the type people gave the critical failure.
"We evaluate category prediction using critical, any-failure, earliest, and terminal accuracy, comparing the predicted category against the annotated critical failure, any annotated failure, the first observed failure, and the terminal failure, respectively." (§4.1)
judge-only baseline (judge with no checks)
The same judge model, given the run and the failure types, but no list of failed checks.
"Table 4 compares AgentRx against a judge-only baseline that sees the trajectory and taxonomy prompt, but no AgentRx-generated validation evidence." (§4.3)
step-by-step generation (writing rules at each step)
Writing new check rules at every step, from only the part of the run up to that step.
"We evaluate two constraint-generation strategies: one-shot, which generates constraints once from the full trajectory, and step-by-step, which generates constraints from each prefix T≤k." (§4.4)
one-shot generation (writing rules once)
Writing all the check rules in one pass, from the whole run.
"We evaluate two constraint-generation strategies: one-shot, which generates constraints once from the full trajectory, and step-by-step, which generates constraints from each prefix T≤k." (§4.4)
average step distance (average steps off)
On average, how many steps the predicted step is from the marked one. Lower is better.
"and average step distance, which measures how far predictions are from the annotated step." (§4.1)
LLM (language model)
A large language model: a model trained on a lot of text that answers in text.
"Large Language Model (LLM)-based agents are increasingly acting autonomously without a human in the loop." (§1)
WebSurfer
The web-browsing agent in Magentic-One.
"It uses an orchestrator to coordinate specialized agents for web browsing, file navigation, and code execution." (§2)
FileSurfer
The file-reading agent in Magentic-One.
"It uses an orchestrator to coordinate specialized agents for web browsing, file navigation, and code execution." (§2)
Magentic-One
A team of agents for open web and file tasks, led by an orchestrator.
"Magentic-One (Fourney et al., 2024) is a generalist multi-agent system for open-ended web and file-based tasks." (§2)
τ-bench
A test set of shop customer-service tasks: one agent, a simulated customer, store tools and a rule book.
"τ-bench (Yao et al., 2024) is a tool-calling benchmark for retail tasks, with an LLM-simulated user and a single agent equipped with domain-specific APIs and policy guidelines." (§2)
Flash
A team of agents that follows a troubleshooting guide for repeat problems in cloud services.
"Flash (Zhang et al., 2024) is a workflow automation agent for diagnosing recurring incidents in production cloud services." (§2)
GPT-4o
An OpenAI language model; it ran the agent in the τ-bench runs.
"We run 115 τ-bench retail tasks using gpt-4o with one trial and analyze the 39 failed trajectories." (§2)
GPT-5
An OpenAI language model; the paper's default model for writing the check rules and for the judge.
"We use gpt-5 as the default model." (§4.1)
DeepSeek-V3.2
An open-weight language model from DeepSeek, tried in place of GPT-5 for the whole pipeline.
"Failure-category accuracy with DeepSeek-V3.2 as the model for both constraint generation and judging." (App. A)
o3
An OpenAI reasoning model, tried as a second judge model.
"o3 judge ablation (mean ± std, n=3; configurations as in Table 4)." (App. A, Table 8)