LongRCA Bench

Pinpointing Error Steps in Long-Horizon Agent Failures.

1,140 trajectories · 5 benchmarks · exact failure-step attribution.

Read the Paper

Download the Dataset

1,140TrajectoriesPublic evaluation set

5BenchmarksDiverse agent settings

145Median stepsLong-horizon traces

48Median root-to-endSteps after the root cause

01 / Paper Results

LongRCA Bench Leaderboard

Rank	Method	Overall	SWE-bench Pro	Terminal-Bench 2	TravelPlanner	VitaBench	WebArena
Loading paper results…

02 / Case Study

An Example of a Failed Task Trajectory

A long execution can report completion while still returning a failed result. This TravelPlanner trace shows how a missed constraint survives verification and reappears in the final answer.

TravelPlanner · Query 44

End-to-End Multi-Agent Execution Trace

ScrantonNewarkMar 24–26, 20221 traveler$1,600 budget

TASK_COMPLETE

77message steps

48tool requests

23 / 8 / 4candidates / checks / notes

1 rework looplodging city mismatch

Message flow across agent lanes

Dispatch Tool call Verification Error / fail Completion

0 / 76

System / Terminal Plan

Loading trajectory…

Candidates: 0
Checks: 0
Notes: 0

Stage-level payloads and returns

Stage	Dispatched payload	Returned evidence / decision	Blackboard delta
P1Plan	User query + grounding workflow	Facts, unknowns, agent assignments	—
P2Transport	Scranton → Newark; Mar 24 / 26; all modes	2 outbound + 1 return flights; taxi quote; no driving record	+4 candidates, +1 note
P3Stay shortlist	Newark; 1 traveler; 2 nights; remaining budget	15 records searched; 4 stored; UWS selected provisionally	+4 candidates, +1 note
P4Food + sights	Newark venues, ratings, prices, addresses	30 restaurants + 20 attractions; 12 strong candidates; $505 trip estimate	+12 candidates, +1 note
P5Verify	20 candidates + 3 notes + trip constraints	5 passes, 2 blocking failures, 1 malformed tool request	+7 check records
P6Stay rework	City mismatch + original 15 lodging records	0 genuine Newark stays; 3 NYC records relabeled with location caveats	+3 candidates, +1 note
P7Decision	Data limitation + $1,315 remaining budget	UWS approved with Manhattan / PATH caveat; TASK_COMPLETE emitted	+1 approval check
P8Writer	WriterPlan schema + full blackboard snapshot	3-day JSON itinerary; approved lodging caveat omitted	Final output

Figure 1. End-to-end execution of TravelPlanner query 44. Each mark is one trace message; vertical movement denotes agent handoff, while the stage ledger records the payload sent and the evidence returned.

Blackboard growth by message step

Final candidate inventory

Transport

Stay records

Restaurants

Attractions

The 7 stay records include 4 initial candidates and 3 location-corrected records added during rework.

Critical correction loop

Initial shortlist15 records labeled Newark
Verifier interceptNames expose Manhattan / Brooklyn
Tool errorStep 55 JSON payload fails to parse
Second search0 genuine Newark stays
Manager decisionAccept UWS with an explicit caveat

Verification conclusion	Status	Effect
Transport dates and prices	Pass	$51–55 round-trip flights; $358 round-trip taxi
Stay occupancy and nights	Pass	1 traveler, 2 nights
Stay city consistency	Fail	Candidates are actually in New York City
Restaurant and attraction city	Pass	6 restaurants and 6 attractions are in Newark
Budget calculation	Pass	$505 estimated; $1,095 remaining
Local constraints	Pass	All constraints are null
Initial overall readiness	Fail	Triggers stay rework
Stay-location exception approval	Pass	Accept UWS Manhattan with a PATH caveat

Budget allocation

Transport $51 Stay $234 Food $180 Attractions $40 Remaining $1,095

Date	Transport	Meals	Attractions	Stay
03-24	F3920949 13:40–14:46	Drifters Cafe Mogambo Khush Hua	Branch Brook Park	1 Bedroom in UWS Manhattan
03-25	–	Drifters Cafe Angeethi Restaurant Bernardo's	Newark Museum of Art Dreyfuss Planetarium Military Park	1 Bedroom in UWS Manhattan
03-26	F3919859 21:46–22:23	Tunday Kababi Artistry	New Jersey Historical Society Newark Riverfront Park	–

The final JSON still labels “1 Bedroom in UWS Manhattan” as Newark and drops the Manhattan / PATH caveat required by the Manager.

03 / Contribute

Submit results through a reviewed pull request.

Community results will be validated automatically, reviewed by maintainers, and added after the PR is merged. Because labels are public, submissions should be treated as reproducibility reports.

01 metadata.json

Method, model, authors, date, code URL, log URL, and reproducibility metadata.

02 predictions.jsonl

Exactly 1,140 rows containing question_ID, predicted_role, and predicted_step.

03 REPRODUCE.md

Environment, prompts, settings, commands, and enough detail to rerun the method.

Keep large artifacts outside the Pages repository.

Store traces and raw logs in durable external storage, then include public links in metadata.json.

Read the submission guide

04 / Citation

Build on LongRCA Bench.

If this benchmark, dataset, or leaderboard supports your work, please cite the paper.

arXiv:2608.15242 cs.AI · cs.SE 2026

View paper on arXiv

@article{zhang2026longrca,
  title   = {LongRCA Bench: Diagnosing Responsible Roles and Root Causes in Long-Horizon Agent Failures},
  author  = {Zhang, Yunfei and Feng, Boyu and Pei, Changhua and Wang, Zexin and Peng, Zhihuang and Liu, Xinlong and Jiang, Hengyue and Ma, Difeng and Zhang, Jiayi and Yao, Yongzhou and Zhao, Yanan and Sun, Fei and Huo, Yintong and Liu, Zhaoyang and Li, Jingjing and Xie, Gaogang and Pei, Dan},
  journal = {arXiv preprint arXiv:2608.15242},
  year    = {2026}
}