IBM Research Shows Agent Harness Choice Swings Task Success from 30% to 81%

IBM Research has published a controlled comparison that should reshape how enterprises evaluate agent frameworks. Using a single open-weight model — GLM-5.3-flash — across 90 AppWorld tasks spanning email, payments, and file-storage APIs, the team swapped only the harness layer. CUGA, IBM's ConfigUrable Generalist Agent, completed 81.1% of tasks. Hermes, built for always-on personal assistants, reached 46.7%. Deep Agents, optimized for long research chains, trailed further behind.

Why the harness matters more than the model

AppWorld acts as a proxy for enterprise automation: it demands login handling, API-document reading, pagination, and multi-app state changes. CUGA's architecture bakes in task decomposition, pre-execution verification, and automatic reflection — guardrails that keep agents inside policy boundaries regardless of model output. Hermes and Deep Agents rely on middleware and retries but lack built-in rule enforcement.

For buyers provisioning GPU clusters, the takeaway is clear: harness choice directly impacts the compute efficiency of every inference dollar. A harness that reduces failed trajectories cuts wasted GPU cycles and shortens time-to-result. CUGA's ability to run inside sovereign or air-gapped clouds also aligns with data-residency mandates that public SaaS agent platforms cannot satisfy.

IBM has open-sourced the evaluation fork (harbor-eval) so teams can reproduce the test on their own hardware. If your roadmap includes putting agents in front of CRM, ERP, or payment systems, benchmark the harness first — before you scale the model.