Operational workflows
Evaluate multi-step processes involving decisions, tool use, approvals, exceptions and changing information—not one-shot prompts.
OPERATEBENCH / RESEARCH PREVIEW 2026
AI can answer.
Can it operate?
The real-world benchmark for AI agents executing complex service operations.
Not another multiple-choice exam. We test whether models can navigate evolving information, tools, approvals and exceptions to complete entire operational workflows—correctly.
21models observed
44synthetic cases / campaign
02metrics · goal & quality
OPERATIONS / SIMULATION● PREVIEW
01 / INTAKERequest receivedContext & constraints
02 / REASONPlan & verifyPolicies · authority
03 / EXECUTETools & actionsApprovals · exceptions
04 / EVALUATEDid it really work?Independent checks
END-TO-END RELIABILITY≠ SELF-REPORTED SUCCESS
01 / THE CORE QUESTION
Passing a prompt is easy.
Running operations is hard.
Real work takes dozens of decisions. A successful agent must identify the right information, respect permissions, handle exceptions and leave the system in the right state. OperateBench evaluates outcomes—not just plausible answers.
The initiative · Proposed 2026 expansion
Can AI agents complete real operational work?
Beyond single prompts and isolated tasks: multi-step service workflows with changing evidence, approvals, policies and real-world constraints.
OperateBench is building toward a cross-industry evaluation of end-to-end agent performance, developed with service companies contributing realistic, privacy-safe synthetic scenarios. Our ambition is to learn which systems reliably complete the job, not just report success.
Benchmark design
Real service operations. Different industries.
Cross-industry cases
Invite service operators and AI-native companies across property management, logistics, banking, legal services and other sectors. Potential contributors are not yet confirmed.
Public + held-out evaluation
Publish a clear methodology and suitable synthetic examples while designing private, held-out tests to reduce benchmark contamination. The held-out program is proposed, not live.
Methodology & governance
Independent scrutiny is part of the plan.
We intend to establish a small methodology board, bringing together project leads and independent researchers. Its remit would include reviewing scenario design, evaluation criteria, scoring, reproducibility and how to communicate limitations. We are seeking an academic or research expert to help validate the approach and potentially co-author a methodological paper.
No independent methodology board, validation or research publication is being claimed for the current preview. Current results are descriptive observations under heterogeneous conditions.
Early contributors · Invite only
Help make the benchmark representative.
We are preparing an invitation-only first cohort of companies transforming service delivery with AI. The goal is a broad range of genuinely complex operational workflows, not a list of company endorsements.
What contributors provide
Representative workflow descriptions and privacy-safe synthetic cases, with expected outcomes and checks. Please do not submit real customer data, secrets or proprietary records to a public repository.
What we provide
Submission guidance and a proposed contribution format, followed by methodology review. Exact requirements, workload and access rules are still being developed.
How to participate
Initial introductions are coordinated privately with the project team. Participation, public credit and publication timing will be confirmed with each company before launch.
Roadmap · Indicative
Working toward a December launch.
Now · OctoberRefine the methodology, identify independent reviewers and shortlist founding contributors.
NovemberOnboard participating teams, expand cases across industries, define held-out checks and prepare a coordinated communications plan.
Target · Early December 2026Publish the expanded benchmark with contributors, methodology and clearly stated limitations, if validation and collection are ready.
Target is the first or second week of December, subject to contributors and methodological readiness—not a committed release date.
02 / THE DATA
Benchmark leaderboard.
Experimental results across 21 models. Quality and goal achievement are separate measurements; neither is a universal capability score.
◎
Strict quality
Percentage of scored cases satisfying all reported quality checks.
↗
Goal achieved
Percentage of scored cases reaching the intended outcome, even if some checks fail.
$
Run cost estimate
Estimated campaign costs—not production pricing or invoices.
Unweighted average of two original per-observation rates, with heterogeneous settings: descriptive, not a controlled ranking. Original reported grades and the known correction are explained by the pinned observation study.
gpt-6-astra96.6%gemini-4-argon95.5%gpt-6.1-sol95.5%claude-fable-5-180.5%claude-opus-5-554.5%gemini-3.8-flash45.3%gpt-6-sol44.3%muse-wonder-1.026.5%zai-org/GLM-5.3-Flash14.8%nvidia/GLM-5.2-NVFP4-262K13.6%muse-spark-1.310.2%Qwen/Qwen3.8-27B-FP83.4%deepseek-ai/DeepSeek-V4.1-Flash2.3%deepseek-ai/DeepSeek-V4-Flash-07312.3%gpt-6-luna0.0%Qwen/Qwen3.6-35B-A3B-FP80.0%nvidia/Kimi-K2.6-NVFP40.0%Qwen/Qwen3.5-397B-A17B-FP80.0%mistral-medium-3-50.0%nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP40.0%Qwen/Qwen3.5-9B0.0%
| Model | Average quality (%) ↓ | Average goal achieved (%) | Average full-run estimate (USD) |
|---|---|---|---|
gpt-6-astra | 96.6% | 100.0% | $238.13 |
gemini-4-argon | 95.5% | 100.0% | $94.57 |
gpt-6.1-sol | 95.5% | 100.0% | $52.67 |
claude-fable-5-1 | 80.5% | 100.0% | $192.63 |
claude-opus-5-5 | 54.5% | 98.9% | $153.68 |
gemini-3.8-flash | 45.3% | 100.0% | $29.09 |
gpt-6-sol | 44.3% | 89.8% | $74.55 |
muse-wonder-1.0 | 26.5% | 56.4% | $52.06 |
zai-org/GLM-5.3-Flash | 14.8% | 77.3% | $1.21 |
nvidia/GLM-5.2-NVFP4-262K | 13.6% | 71.6% | $1.73 |
muse-spark-1.3 | 10.2% | 42.0% | $46.82 |
Qwen/Qwen3.8-27B-FP8 | 3.4% | 3.4% | $2.52 |
deepseek-ai/DeepSeek-V4.1-Flash | 2.3% | 18.4% | $2.90 |
deepseek-ai/DeepSeek-V4-Flash-0731 | 2.3% | 5.7% | $2.54 |
gpt-6-luna | 0.0% | 21.5% | $1.42 |
Qwen/Qwen3.6-35B-A3B-FP8 | 0.0% | 17.0% | $1.96 |
nvidia/Kimi-K2.6-NVFP4 | 0.0% | 16.1% | $1.00 |
Qwen/Qwen3.5-397B-A17B-FP8 | 0.0% | 6.8% | $1.36 |
mistral-medium-3-5 | 0.0% | 0.0% | $25.77 |
nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 | 0.0% | 0.0% | $2.26 |
Qwen/Qwen3.5-9B | 0.0% | 0.0% | $1.27 |
Known 44-case campaign estimates including known retries, not invoices. Some estimates have incomplete recorded usage. River uses owner-selected $0.12/$0.01/$0.50 per million input/cached-input/output tokens; ranges compare hypothetical all-cached versus no-cache usage. Other estimates retain campaign tariffs.
About OperateBench
OperateBench explores whether agents can keep operations safe and progressing as time, evidence and authority change. These exploratory observations cover related synthetic cases; the official Lifecycle Track remains a technical preview. Read about the project ↗