IBM's CUGA Harness Lifts the Same Open Model From 30% to 81% on AppWorld While Policies Stay Outside the Prompt

IBM Research has published a controlled comparison that keeps the model fixed and changes only the agent harness—and reports task completion swinging from 30% to 81.1% on the same 90 AppWorld tasks. The November 3, 2026 Hugging Face post argues that enterprise automation needs a harness built for multi-app API work and for rules that hold even when the model is wrong. CUGA (ConfigUrable Generalist Agent), IBM’s Apache-2.0 open harness, is the system under that claim.

The experiment used GLM-5.3-flash for every run, one attempt per task, across 90 tasks in 30 scenarios, scored with the Harbor evaluation framework. CUGA’s AppWorld configuration—task decomposition, verification before execution, and automatic reflection—reached 81.1% task goal completion (73 of 90) and 66.7% scenario goal completion (20 of 30). Harbor’s default Hermes setup scored 46.7% / 26.7%. Deep Agents with AppWorld tools over MCP plus LangChain middleware scored 30.0% / 6.7%. IBM says Hermes and Deep Agents were not built for AppWorld-style multi-app API work, so the gap is a fit story, not a universal ranking. The setups were not identical, policies were off during the bake-off, and the sample is one benchmark—limits the post states up front. The evaluation fork lives in harbor-eval so others can rerun configurations.

What the harness owned on every loop

Every CUGA agent runs a compact prepare–model–execute loop that scales from a single CugaAgent to a CugaSupervisor coordinating local or remote agents over A2A. In the AppWorld configuration, planning splits work one app at a time and shows each step only that app’s tools plus a short shared set; a find_tools shortlister can pull anything else. Action uses CodeAct: the model writes Python that calls APIs inside a time-limited sandbox. Before data-changing code runs, a verification pass reviews the block. After execution, automatic reflection can revise the plan, and a pagination audit flags list calls that never requested the next page. Those steps are where the model makes fewer free choices—and where a policy can later sit.

Policies that stop a call, not just advise it

Most teams put payment limits and PII rules in the system prompt. CUGA’s policy layer, described in IBM’s ACM CAIS 2026 paper Governance by Construction for Generalist Agents, moves those rules into the harness at four points: an Intent Guard that can block or answer a request before the agent acts; Playbooks that prescribe a procedure; Tool Approval that pauses for a person and shows the proposed code; and Tool Guides with ToolGuards that run hard checks so a blocked tool is never called. An Output Formatter can force an approved reply template. Approvals and ToolGuards are runtime stops; prompt guidance is not. In the Policy Playground demo IBM describes, the same “Pay everything I owe” message led an unguarded agent to pay $90 including a denied claim, while the policy agent held the payment and paid only the $10 due.

Why this is an enterprise pick, not a coding REPL

CUGA is positioned for CRM, payments, and similar business systems under sovereign deployment: local runs, Kubernetes in a private or sovereign cloud, or air-gapped clusters, with OIDC and separate chat versus management roles. It works with any OpenAI-compatible endpoint, including open-weight models. Budgets and per-run receipts expose tokens, tool calls, and time. Configuration lives in the UI or SDK—draft, test, publish—rather than only in application code. The public repository cuga-project/cuga-agent is Apache-2.0; IBM notes hosted CUGA held long #1 stretches on WebArena and AppWorld, separate from the open-weight bake-off above.

The same post is clear about when not to choose it: terminal coding agents belong in opencode, Codex, or Claude Code; always-on personal assistants in OpenClaw or Hermes; blank-canvas graphs in LangGraph directly. CUGA’s bet is narrower and more operational—automate business work across APIs, apps, and web UIs under rules the harness can enforce, and measure what the harness itself contributes when the model stays the same.