Ox Alpha Software Benchmarks Show 62.8% DeepSWE Score Following Initial 80% Sample

Independent evaluations of Ox Alpha show the artificial intelligence model achieved a 62.8% completion score across the full DeepSWE benchmark, following an earlier 10-task test that reported an 80% success rate. The larger evaluations place Ox Alpha’s coding performance near DeepSeek V4 Pro, while Chinese AI company Z.ai has officially identified the system as its open-weight model GLM-5.3-Flash under the MIT license.

Initial DeepSWE Claims and Sample Size Limitations

The initial benchmark claim for Ox Alpha circulated after developer Ben Davis ran the model on a sample of 10 tasks selected from the DeepSWE software-engineering benchmark. Ox Alpha resolved eight of those 10 tasks, producing a reported score of 80%.

The initial report contrasted that figure against scores associated with other frontier models:

• Ox Alpha: 80%
• Claude Fable 5: 65%
• GLM-5.3: 62%
• Grok 4.6: 62%
• GPT-5.6 Sol: 52%

The comparison paired a 10-task sample of Ox Alpha against benchmark figures derived from broader evaluation sets. Because the DeepSWE benchmark comprises 113 tasks, each individual problem in a 10-task run shifts the overall score by 10 percentage points.

Davis later clarified that subsequent, larger evaluations provided a more representative measurement of the model’s performance.

Full-Benchmark and Subset Evaluation Results

Following the initial test, Wenqi Huang, a researcher at DataCurve who contributed to the creation of DeepSWE, evaluated Ox Alpha on an expanded subset of the benchmark. Ox Alpha achieved a score of approximately 63% in that run, consuming an average of roughly 47,000 output tokens per task.

A separate community evaluator subsequently ran Ox Alpha across the complete 113-task DeepSWE dataset, reporting a final score of 62.8%. During that complete run, two agent instances hit the benchmark’s 90-minute execution limit.

The community evaluation was conducted as a single pass without the repeated trials or official confidence intervals used for formal leaderboard submissions.

Official DeepSWE Leaderboard Standings

As of August 24, 2026, Ox Alpha does not appear on the official DataCurve DeepSWE leaderboard.

The top recorded scores on the verified leaderboard include:

• Claude Opus 5 (maximum reasoning): 74% ±4%
• GPT-5.6 Sol (maximum reasoning): 73% ±3%
• Claude Fable 5 (maximum reasoning): 70% ±4%
• GLM-5.3 (maximum reasoning): 69% ±3%
• Kimi K3 (maximum reasoning): 69% ±5%
• GPT-5.6 Luna (maximum reasoning): 67% ±4%
• GPT-5.5 (extra-high reasoning): 67% ±6%
• Grok 4.6 (extra-high reasoning): 67% ±2%
• Gemini 3.7 Flash (high reasoning): 65% ±2%
• DeepSeek V4 Pro (maximum reasoning): 63% ±6%

Ox Alpha’s reported 62.8% full-run community mark aligns closely with DeepSeek V4 Pro’s 63% result, trailing the leading configurations from Anthropic, OpenAI, Z.ai, Moonshot, xAI, and Google.

DeepSWE Architecture and Requirements

DeepSWE evaluates AI agents on long-horizon, autonomous software engineering assignments rather than isolated function-generation problems. Agents interact directly with real codebases in isolated execution environments to resolve issues.

The benchmark contains:

• 113 original engineering tasks written from scratch to prevent training-data contamination
• 91 active open-source repositories
• Codebases across TypeScript, Go, Python, JavaScript, and Rust
• Programmatic verification systems to test submitted fixes

To complete an assignment, an agent must locate relevant files across the repository, implement code modifications across multiple components, execute the test suite, resolve errors, and output a functional patch.

Terminal-Bench 3.0 and EVMBench Results

Community evaluators have tested Ox Alpha on two additional technical benchmarks: Terminal-Bench 3.0 and EVMBench.

On Terminal-Bench 3.0, which tests an agent’s ability to operate inside a command-line environment, Ox Alpha scored 26.6% on a single-pass run. That result placed the model between GLM-5.3 and Grok 4.6, above Claude Opus 4.8 and below Claude Fable 5. The evaluator excluded two benchmark tasks because of local hardware constraints on required H100 GPUs, and the run did not use the five-trial averaging protocol standard on the official leaderboard.

On EVMBench, an evaluation measuring smart-contract security audit performance across 12 frozen audits, Ox Alpha recorded a 37.2% micro recall and 80% precision, identifying 16 out of 43 identified vulnerabilities. It posted a median completion time of 52 seconds per audit.

The EVMBench test yielded the following comparative results across tested systems:

• GPT-5.6 Sol: 40.3% micro recall
• Ox Alpha: 37.2% micro recall
• Grok 4.6 Build: 36.4% micro recall
• GLM-5.3: 11.6% micro recall

The EVMBench evaluation was conducted by a single reviewer using a single run per audit. Ox Alpha was officially revealed and claimed by Chinese AI company Z.ai as its open-weight model GLM-5.3-Flash, released under the MIT license.