Once the AI leaderboard was recalibrated, everyone’s scores dropped.

One Recalibration, and Every Model Lost 6 to 15 Points

As of July 31, 2026, a recently published audit of an AI leaderboard has sparked discussion among developers: after its maintainers reexamined and recalibrated the scoring scale, every model on the leaderboard lost between 6 and 15 points.

The models themselves had not been updated. The test questions had not suddenly become harder, nor had inference services collectively degraded. The change came from the ruler—the scale used to map model capabilities to scores may have been too lenient all along.

The most concerning part is not that any particular model lost a few points, but that the industry has long treated evaluation scores produced through multiple layers of transformation as capability metrics approaching objective fact. If a single recalibration can knock every contender down an entire tier, then many leaderboard claims—such as “85 points,” “close to human experts,” or “10% ahead of the previous generation”—may contain a substantial degree of measurement illusion.

More precisely, this change does not prove that all AI evaluations are wrong, nor does it mean that model capabilities were exaggerated by the same amount. It proves something else: absolute scores on leaderboards are far less absolute than they appear.

The Models Did Not Change, So How Could Their Scores Suddenly Drop by More Than Ten Points?

From prompt to final leaderboard, a model typically passes through question sampling, output generation, answer evaluation, subscore weighting, missing-value handling, and scale mapping. The final number developers see is often not as simple as an “accuracy rate,” but the product of an evaluation pipeline.

A typical process can be simplified as follows:

1. Have different models complete the same set of tasks;
2. Use rules, unit tests, or another large model as the judge;
3. Combine raw results across multiple dimensions into a total score;
4. Map the raw score onto a more communicable scale, such as 0 to 100;
5. Rank models by total score and attach labels such as “excellent” or “expert-level.”

The problem often arises in step four. Suppose a group of models has raw capability values concentrated between 0.42 and 0.68. To create greater visual separation, a leaderboard might use linear scaling to display them as scores ranging from 65 to 95. If the maximum, minimum, or reference anchors are set improperly, the entire leaderboard will look more optimistic than reality warrants.

It is a bit like converting an exam with an average accuracy rate of only 60% into an average score of 80. The ranking among students may still be correct, but the interpretation that “80 means a good command of the material” has already become distorted.

Recalibration may also expose the following problems:

• Weak benchmark anchors: Using an older, limited model as the low-end reference inflates the normalized scores of newer models;
• Insufficient compression at the high end: Models near the upper bound receive overly optimistic scores, making all of them appear close to perfect;
• Overly lenient judge models: LLM judges favor fluent, comprehensive, and attractively formatted answers while overlooking factual errors;
• Failures penalized too lightly: Timeouts, tool-call failures, empty answers, or non-executable code are not penalized according to their actual business cost;
• Distorted sample distribution: Easy questions are overrepresented, while a small number of critical hard questions are drowned out by the average;
• Unaddressed version drift: Scores from before and after changes to the dataset, system prompt, judge model, or inference parameters are still compared on the same scale.

Therefore, “every model lost 6 to 15 points” is better understood as a correction to the measurement system than as a collective decline in model capabilities.

The Ranking May Be Right, but the Product Decision May Already Be Wrong

If every model lost exactly the same number of points, the ranking would not change. Some might therefore conclude that this is merely a presentation-layer adjustment with no practical impact.

That conclusion is too optimistic.

Developers do not use leaderboards solely to identify the top-ranked model. They also use absolute scores to make product decisions: whether a model meets the threshold for deployment, whether it can replace human review, whether a more expensive inference tier is worth paying for, and whether a particular capability has been “mostly solved.” If the scale is inflated, all of these judgments can be skewed.

For example, if a coding agent scores 88 on a leaderboard, a team may believe it can autonomously modify production code. If recalibration reduces that score to 74, the implication is entirely different. The former suggests occasional human confirmation; the latter may still require mandatory review, rollback mechanisms, and comprehensive testing.

The problem becomes more serious when score changes are uneven. The audit reported declines ranging from 6 to 15 points, indicating that different models were affected differently. Even if the top rankings were not completely reordered, the gaps between models, cost-performance conclusions, and capability tiers may all have changed.

A leaderboard should therefore answer at least three questions:

• Which model is better than which;
• By how much, and whether the difference exceeds statistical error;
• What the score actually means in real-world tasks.

Many current AI leaderboards can barely answer the first question, yet they use numbers precise to decimal places to create the impression that the other two have also been resolved.

Inflated Evaluations Are Not Just a Normalization Formula Problem

The scoring scale is only the surface-level issue. The deeper problem is that we often measure “whether a model’s response resembles a high-scoring answer” rather than “whether the model can complete the task reliably.”

LLM Judges Reward Models That Know How to Present Answers

When the judge is also a large model, long responses, clear headings, citation-like phrasing, and a confident tone may all confer an additional advantage. Even when an answer is factually wrong, it may defeat a correct but concise answer if it is well structured and fluently explained.

This bias is especially visible in open-ended question answering and anonymous arenas. Users often lack the time to verify every factual claim, so voting can easily become an immediate judgment of presentation style. The resulting leaderboard may measure “which answer looks better at first glance” rather than “which answer is demonstrably better after verification.”

Static Question Banks Are Becoming Training Material

Classic benchmarks such as MMLU and HumanEval were once extremely useful, but after their questions remain public for long periods, model teams can optimize around question types, answer patterns, and high-scoring solution paths. Even if test answers are not inserted directly into the training set, related solutions, discussions, and variants may still enter the pretraining corpus.

This is similar to students practicing every question in an exam bank beforehand: a high score still reflects some ability, but it is no longer equivalent to transfer performance in unfamiliar scenarios.

Longer Reasoning Does Not Necessarily Produce Better Answers

Recent open-ended computer science evaluations also show that increasing reasoning effort does not always improve performance monotonically. According to the referenced material, in one controlled experiment, GPT-5 Thinking’s average score rose from 7.9 to 15.3 when switching from low to medium reasoning effort. When reasoning effort was increased further to high, however, the score fell to 12.6.

These results are more informative than saying that “the model gained a few more points.” They suggest that current reasoning models are good at identifying textbook-style question types and applying established patterns, but when faced with tasks that lack standard answers and require interactive queries or continuous optimization, additional computation may merely make the model more committed to a suboptimal path.

In other words, traditional benchmarks often reward “finding a correct answer,” while real-world engineering requires “finding a better answer within cost constraints.” These are not the same thing.

How Developers Should Redesign Their Own Evaluations

For ordinary development teams, the value of this audit is not that it tells everyone to switch to a different leaderboard. It is that it reminds teams to treat internal evaluation as a software system requiring version control and continuous calibration.

1. Preserve Raw Results, Not Just Aggregate Scores

At a minimum, retain the input for each case, the model output, inference parameters, latency, cost, evaluation result, and failure reason. Otherwise, whenever scoring rules change, every model must be rerun, which is both expensive and makes historical results difficult to reproduce.

A minimal evaluation record might use a structure like this:

{"case_id":"refund_042","model":"model-a-2026-07","prompt_version":"v3","judge_version":"rule-2.1","passed":false,"failure_type":"fabricated_policy","latency_ms":1840,"input_tokens":923,"output_tokens":411,"cost_usd":0.012}

The most important field here is not the aggregate score, but failure_type. In a production system, fabricating a refund policy and omitting a Markdown heading must not both be represented as the same “one-point deduction.”

2. Incorporate Business Losses Into the Scoring Function

Different errors carry different costs. A customer service model that sounds slightly unnatural usually creates only a user-experience issue. Fabricating contract terms, leaking private information, or invoking the wrong tool may constitute an unacceptable incident.

Average accuracy should therefore not be the only metric. A more practical evaluation should at least separate:

• Critical-task success rate;
• Severe error rate;
• Tool-call success rate;
• P50 and P95 latency;
• Cost per successfully completed task;
• Human takeover rate;
• Multi-turn task completion rate.

If Model A has an overall accuracy rate two percentage points higher than Model B but three times the severe error rate, the vast majority of production teams should not choose Model A.

3. Add Confidence Intervals to Scores

Leaderboards based on small samples are especially prone to creating false precision. A score of 81.7 is not necessarily truly better than 80.9; the difference may simply reflect sampling variation.

Teams can use repeated sampling, bootstrapping, or runs with multiple random seeds to report confidence intervals alongside means. For nondeterministic models, the same prompt should be run multiple times. If the intervals for two models overlap substantially, the conclusion should be “currently indistinguishable,” not a forced first- and second-place ranking.

4. Rotate Judges Regularly and Conduct Spot Checks

An LLM judge is not objective truth; it is another model with its own preferences. Teams should combine rule-based evaluation, executable tests, multiple judge models, and blinded human review, while regularly checking the agreement rate among them.

Three types of samples deserve particular scrutiny: answers that judges rate with high confidence, answers on which different judges disagree, and stylistically strong answers that lack verifiable evidence. The truly dangerous outputs are not obvious nonsense, but errors written to look convincingly correct.

5. Build Dynamic Test Sets While Retaining Fixed Anchors

Test sets should continuously incorporate real production failures to prevent models from showing false improvement merely by adapting to static question banks. At the same time, some stable anchors should be preserved to track long-term changes in capability.

Using only fixed questions invites contamination and leaderboard gaming; using only new questions makes longitudinal comparison impossible. A more reasonable combination is “fixed regression set + rolling challenge set + production shadow traffic.”

Scenario-Specific Leaderboards Are More Useful Than Overall Rankings

What developers actually need is usually not “the strongest model in the world,” but “the most suitable model under my constraints.”

Models such as Claude may perform consistently in real-world software engineering, code maintenance, and long-context reading, yet may not excel at finding algorithmically optimal solutions. Other reasoning models may lead on mathematics problems, but their latency, price, and tool-use reliability may make them unsuitable for online services. Compressing these differences into a single overall score is itself a loss of information.

More useful leaderboards should be divided by scenario: code repair, algorithm optimization, retrieval-augmented question answering, customer service, multi-turn agents, structured extraction, and safety review should all be evaluated separately. Even when an overall score is necessary, the weighting, raw scores, sample size, confidence intervals, model version, and evaluation date should be disclosed.

The same model name does not imply constant capability. Providers may update backend weights, system prompts, safety policies, and reasoning budgets. A score without a date and version number may have a shelf life of only a few weeks.

The Score Drop Actually Makes the Leaderboard More Credible

A collective decline of 6 to 15 points is not flattering, but proactively auditing and correcting the scale is more valuable than preserving attractive numbers. The truly untrustworthy leaderboard is not one that revises its scores, but one whose scores only ever rise and never fall, even when its methodology changes without historical results being recalculated.

However, merely remapping the scale is not enough. Maintainers should also publish the basis for recalibration, the raw distribution, anchor selection, historical versions, and error ranges so that outsiders can determine whether the adjustment corrects bias or merely replaces it with a different but equally arbitrary algorithm.

The conclusion for the industry is straightforward: model leaderboards can be used to identify candidates, but they cannot replace product-specific evaluations; relative rankings can serve as references, but absolute scores require scrutiny of their definitions; and any model described as “close to perfect” should first be retested against real-world failure cases.

AI models did not collectively become less intelligent overnight. Once the ruler was straightened, we could finally see how far they still are from the perfect scores promised in the marketing.

References

• Zhihu: Why Evaluating Today’s Large Models Is Becoming Increasingly Difficult: Introduces the role of large-model benchmarks, capability boundaries, and new approaches to highly challenging evaluations, helping explain why static leaderboards can easily become ineffective.