Amazon Releases SOP-Bench to Evaluate AI Agents on Industrial Operating Procedures

Amazon researchers introduced SOP-Bench on August 21, 2026, an open-source evaluation benchmark designed to test AI agents against standard operating procedures across 12 business domains. Presented at the KDD 2026 conference, the framework contains over 2,000 executable tasks paired with functional tool interfaces and verified ground-truth answers, evaluating agents on procedural execution rather than text-only responses across 11 frontier foundation models.

Benchmark Architecture and Supported Domains

SOP-Bench converts written operational guidelines into runnable computational environments. According to the research team, led by Amazon Artificial General Intelligence engineer Rohith Nama and Amazon Store senior staff scientist Nandi Subhrangshu, the dataset spans 12 operational sectors, including:

• Healthcare patient intake
• Dangerous-goods shipping classification
• Customer service interactions
• Trust and safety content moderation
• Financial compliance reviews
• Warehouse facility inspections
• Video annotation workflows
• Inbound email intent triage

Each procedure within the benchmark is structured around four primary components: the raw standard operating procedure text, callable tool functions, technical specifications for those tools, and a test suite containing verified ground-truth outcomes.

Unlike benchmarks that evaluate model output using automated text scoring or isolated API calls, SOP-Bench executes an agent’s actions through mock APIs and deterministic tooling. Agent trajectories are graded using three execution-oriented metrics:

• Execution Complete Rate (ECR): The proportion of runs where the agent reaches an endpoint without runtime crashes or infinite loops.
• Completed-Task Success Rate (C-TSR): The accuracy rate calculated strictly among runs that finished execution.
• Task Success Rate (TSR): The overall percentage of tasks executed correctly relative to the entire evaluation set.

Dataset Construction and Expert Verification

The benchmark was built using a hybrid workflow combining domain professionals and large language models. Human industry experts authored the original operational procedures and defined the decision parameters.

Anthropic’s Claude 3.5 Sonnet v2 model was then used to convert the expert text into executable tasks. The model generated the underlying data schemas, mock API interfaces, tool definitions, executable Python code, and evaluation datasets containing standard inputs, edge cases, and deliberate operational failure states.

Following model generation, the human domain experts reviewed every component to verify procedural logic, validate tool outputs, correct errors, and execute test runs. No proprietary or sensitive enterprise data was used during the construction of the benchmark.

Evaluation Results Across Frontier Models

The researchers evaluated system performance across 11 frontier foundation models using two primary agent architectures: a reasoning-style ReAct agent and a function-calling agent. The ReAct agent was evaluated across all 11 frontier models, whereas the function-calling agent was tested exclusively on Claude models due to its dependence on native tool-use APIs.

The evaluation revealed several distinct operational behaviors:

• Model upgrades did not guarantee higher success: In tests using the reasoning agent setup, the newer Claude 4.5 model family recorded lower average task success rates than the older Claude 4 family.
• Distractor tools degraded execution: In a video-annotation task requiring six specific tools, adding 20 plausible but non-functional tools to the agent’s environment reduced the success rate by nearly half.
• Task difficulty varied significantly: Agent performance ranged from approximately 90% accuracy on simpler workflows, such as inbound email classification, to roughly 25% accuracy on complex tasks, such as multi-object driving video annotation.
• Agent architecture influenced speed and accuracy: Across 13 head-to-head procedural tests on the same underlying model, the ReAct reasoning agent scored higher in eight tests and achieved a slightly higher average score than the function-calling agent, but required roughly 33% more execution time per task.
• Reading volume reduced performance: Procedures requiring prolonged text comprehension with few decision points produced lower agent accuracy than shorter procedures containing frequent branching logic.

Baseline Agents and Open-Source Availability

Amazon released the complete SOP-Bench suite publicly on GitHub and Hugging Face. The release includes the 12 expert-authored procedures, the mock tools and task datasets, execution tracking scripts, and two reference baseline agents.

The evaluation runner logs full execution trajectories, tool invocations, and internal reasoning steps, allowing developers to trace procedural failures back to specific steps. The research team outlined technical roadmaps to expand the repository with multimodal procedures containing images and tables, nested procedures requiring mid-task context switching, and additional operational domains contributed by outside developers.