Code Model Rankings, Rewritten by Agents

SWE-ReBench recently released a new round of comparative evaluations targeting real-world software engineering tasks: 13 models and 4 Agents across five programming languages—Go, Java, Python, Rust, and TypeScript.

What makes this evaluation noteworthy is not merely the larger number of participating models, but the fact that it no longer treats the “model score” as the sole conclusion. It places models, Agent scaffolds, and programming languages within the same coordinate system, attempting to answer a question much closer to the real choices developers face: If the same model is paired with a different Agent, or moved from a Python project to a Go, Java, or Rust project, can it maintain its capabilities?

The answer is not encouraging. Model performance in code evaluations is far less stable than a single percentage on a leaderboard suggests. An Agent’s tool design, context management, command execution strategy, and stopping criteria can all change the final result. Differences among languages in compilers, package managers, and testing ecosystems can further amplify these variations.

What Exactly Was Evaluated?

SWE-style tasks do not ask a model to complete a function, nor are they competitive programming problems. An Agent receives an issue description from a real code repository, enters an isolated environment, and autonomously reads code, searches for references, executes commands, modifies files, and runs tests. A submitted patch counts as solving the task only if it passes the required tests.

The entire process can be simplified into the following evaluation matrix:

Final result = Model capability × Agent scaffold × Language ecosystem × Repository structure × Runtime budget

Model: Understand the issue, reason, and generate a modification plan
Agent: Search files, invoke tools, maintain context, and decide when to stop
Language: Type system, compiler feedback, dependencies, and testing toolchain
Task: Real-world issue, cross-file changes, and regression tests
Budget: Time, tokens, number of calls, and reasoning cost

SWE-ReBench covers 13 models and 4 Agents in this round, but “coverage” does not necessarily mean that every model, Agent, and language forms a fully balanced Cartesian product. When reading such leaderboards, developers must still check the number of tasks in each cell, missing combinations, and runtime budgets rather than drawing conclusions solely from the overall ranking.

The choice of five languages is also grounded in practical considerations: Python is the most mature language in the existing SWE-bench ecosystem and the one to which models have had the greatest training exposure; Java and Go represent enterprise backend and cloud-native projects; TypeScript represents frontend and Node.js engineering; and Rust confronts Agents with types, lifetimes, concurrency, and compiler-error handling all at once.

In other words, this is not the same algorithm problem translated into five different syntaxes. Each language comes with different repository structures, build tools, and failure modes.

Rankings No Longer Belong to Models Alone

The most important signal from this comparative evaluation is that it further undermines the simplistic assumption that “a model with a high SWE score must be good at coding.”

In traditional model evaluations, the model is often the only variable: give each model the same prompt and compare whether its output is correct. But in software engineering Agents, the model does not submit its answer directly. It must operate through a layer of scaffolding that determines which files the model sees, which tools it can use, how its history is compressed, and whether it is allowed to keep trying after a failure.

An Agent can affect results in at least four areas:

1. Scope identification. Some Agents begin with repository-wide searches and then read a small number of critical files; others quickly open many files, consume their context window, and only then begin making changes.
2. Use of feedback. Compilation failures and test logs are often lengthy. Whether the Agent can extract key errors, preserve stack traces, and return them to the model directly affects the second round of modifications.
3. Modification strategy. Some scaffolds tend to generate a complete patch, while others allow the model to make a series of incremental edits. The former is faster, while the latter is generally better suited to complex repositories.
4. Stopping criteria. Passing tests does not necessarily mean the task is complete, and failing tests does not necessarily mean the Agent should exit immediately. Both premature stopping and unproductive loops reduce Pass@1.

An Agent is therefore not a neutral shell wrapped around a model. It is more like a race car’s chassis and transmission. Put the same engine in different vehicles, and the lap times will differ.

Recent domestic experiments evaluating independent Agent harnesses provide an even more direct example: after fixing the model, tasks, prompts, and runtime environment, merely replacing the harness can still produce double-digit percentage-point differences in Pass@1. For some models, the gap can even exceed twenty percentage points. This means that many capability improvements described as “model upgrades” may actually include scaffold optimizations, budget changes, or expanded tool permissions.

By displaying four Agents side by side, SWE-ReBench at least makes this coupling easier to observe. For developers, this is more useful than adding yet another overall model leaderboard.

High Python Scores Cannot Be Directly Extrapolated to Production Stacks

Over the past two years, SWE-bench has become a standard metric for code model releases, but it has long centered on Python repositories. Python’s advantages include not only simple syntax but also high model familiarity, fast test startup, intuitive error feedback, and the inclusion of many open-source repositories in training data.

Real enterprise projects, however, are rarely written entirely in Python.

Java Agents need to understand Maven or Gradle, module boundaries, generics, and extensive type hierarchies. Go projects often involve interface implementations, module dependencies, concurrency, and table-driven tests. TypeScript tasks combine compiler configuration, runtime behavior, frontend build pipelines, and type declarations. Rust requires changes to satisfy both business semantics and the borrow checker. Even when a model knows “how the issue should be fixed,” it may fail to produce a mergeable patch because it does not handle the toolchain correctly.

This is why multilingual evaluation is more valuable than simply expanding the Python task set. It evaluates not only syntax transfer, but also whether a model can adapt to different engineering ecosystems.

Existing SWE-bench Multilingual data has shown that models generally achieve higher resolution rates on the Python benchmark than on multilingual tasks, with clear gaps among languages. However, developers should not interpret a language’s average score as meaning that “this language is more difficult.” Task sampling, repository size, test stability, and issue type all affect the results. A higher score for Rust than for Go does not mean Rust is inherently more suitable for Agents; it may simply mean that the two task sets differ in difficulty.

A genuinely meaningful comparison should satisfy all of the following conditions:

• Use the same task subset and container images;
• Use the same prompts and tool permissions;
• Provide the same timeout and call budget for each task;
• Fix the model version, inference parameters, and context length;
• Report the sample size for each language rather than only a macro-average;
• Preserve failure trajectories to distinguish failures in localization, modification, and testing.

If these conditions are not fixed, small leads on a leaderboard are usually not statistically meaningful.

Benchmarks May Also Let Agents “Peek at the Answers”

Multilingual SWE evaluations have another easily overlooked problem: the data and containers themselves may leak future information.

Recent inspections of SWE-bench-Multilingual found that, for some instances, Git history after the base_commit remained visible. If an Agent can execute git log, git show, or inspect remote references, it may be able to see the actual fix commit. In that case, even if the tests pass, the evaluation no longer measures code reasoning ability; it measures whether the Agent discovered that the answer was hidden in the repository history.

This type of issue is particularly dangerous for Agent benchmarks. Standard question-answering evaluations only need to control the prompt, but Agents can access the file system, Git metadata, caches, dependency packages, and network resources. If any of these boundaries is not properly sanitized, “finding a leakage path” may be incorrectly counted as “solving a software issue.”

Therefore, in addition to publishing Pass@1, new leaderboards should explain:

• Whether commits and references after the base_commit have been removed;
• Whether external network access and code search are disabled;
• Whether Docker images contain traces of the gold-standard patch;
• Whether test files are fully visible to the Agent;
• Whether failed tasks are reproducible and whether tests are flaky;
• Whether the same model is run multiple times to estimate random variation.

The value of SWE-ReBench ultimately depends not only on how many models it covers, but also on whether these evaluation conditions are transparent and reproducible. The number of models is eye-catching, but benchmark cleanliness is what determines whether the scores are trustworthy.

How Developers Should Read This Leaderboard

For teams preparing to purchase or integrate a code model, the correct way to use this comparative evaluation is not to “simply choose the top-ranked model,” but to first slice the results according to their own technology stack.

If a team primarily maintains Java services, it should prioritize the success rate, average cost, and failure reasons for the Java subset rather than the aggregate score across five languages. If the goal is Rust infrastructure development, the team should also examine whether the Agent repeatedly struggles with the compiler and ultimately relies on local patches that “happen to pass the tests.” Frontend teams should verify whether the TypeScript tasks cover real build pipelines rather than only small type errors in standalone npm packages.

Second, models and Agents should be selected as combinations. A model leading under Agent A may not remain ahead when integrated into a team’s internal scaffold. Different models are particularly sensitive to tool-call formats, long-context compression, and error feedback, so the optimal combination is often not “the strongest model + the most complex Agent.”

Cost cannot be omitted either. Software engineering tasks usually require multiple rounds of searching, editing, and testing, so per-call pricing accounts for only part of the total cost. A cheaper model may require more iterations, resulting in higher total expense and latency. An expensive model that localizes issues more quickly may actually offer a better cost per completed task.

A more robust internal validation process is:

1. Sample 30 to 100 tasks from issues the team actually fixed over the past six months;
2. Remove fix commits and external answers, and build reproducible containers;
3. Fix the budget and conduct blind tests of candidate “model × Agent” combinations;
4. Record elapsed time, tokens, tool-call count, and patch size in addition to pass rate;
5. Manually review patches that pass the tests to exclude test overfitting and destructive changes.

Public benchmarks are useful for narrowing the candidate pool, but they cannot replace internal replay testing. This is particularly true in platform environments where a single API key can switch among multiple mainstream models, including OpenAI-compatible aggregation services such as OpenAI Hub. The real value lies not in constantly chasing the top spot on the overall leaderboard, but in rapidly rerunning horizontal comparisons with the same tasks and Agent.

One Leaderboard Begins to Reveal Real Coding Capability

This SWE-ReBench comparison of “13 models, 4 Agents, and five languages” does not solve every problem in code Agent evaluation. It still focuses primarily on issue resolution within a single repository and remains far removed from real engineering tasks such as dependency migrations, cross-repository retrieval, and building systems from design documents.

But the direction is right: coding capability must move beyond scores for a single model, language, and scaffold toward compositional evaluation.

For model vendors, this means they can no longer simply choose the most favorable Agent, report the highest score, and obscure the scaffold and budget. For Agent developers, the harness is no longer merely an implementation detail, but a product capability that can be independently optimized and measured. For ordinary development teams, leaderboard rankings are becoming less important, while technology-stack compatibility, cost, and reproducibility are becoming more important.

Past coding leaderboards were effectively asking, “Who is best at writing Python?” This comparative evaluation begins to ask a more practical question: “Once placed in different toolchains and real repositories, who can still get the job done?”

The latter question is much closer to the production value of Code Agents.

References

• Juejin: Moving Beyond SWE-bench’s Score-Only Mindset with a Multilingual Benchmark That Independently Measures Harnesses—Introduces a method for comparing different Agent harnesses while holding models and tasks constant, and discusses multilingual subsets, cost controls, and Git history leakage.