Guess K times first, then train the model to identify the optimal solution.

Train Models Without Requiring Them to Get It Right on the First Try

Recently, developer Alexi Glad proposed a highly practical training perspective: instead of requiring a model to make only one prediction for each problem and holding it accountable for that single result, allow it to generate K candidate answers and learn from the best one.

This can be called exploratory modeling. Rather than optimizing conventional one-shot accuracy, it optimizes the model's ability to find a good answer within a limited search budget.

For today's reasoning models, this distinction is crucial. Tasks such as mathematical proofs, code generation, and agent planning often do not have a smooth, deterministic solution path. A model's first sample may take the wrong branch, while a second attempt using a different decomposition may produce the correct result. Traditional training directly penalizes the first failure; exploratory training instead acknowledges a fact: generation itself is a form of search, and a model's capability is reflected not only in its first answer, but also in how many valuable paths it can cover.

The method is not mysterious. It can even be roughly summarized in one sentence:

Sample K times for the same input, use a verifier to identify the best candidate, and concentrate the training signal on the paths that are genuinely valuable.

It looks like Best-of-N inference moved into the training stage, but the truly difficult part is not generating a few more answers. It is defining what "best" means and correctly assigning the set-level outcome to each trajectory.

From pass@1 to best-of-K

Traditional supervised fine-tuning typically optimizes the likelihood of a reference answer. Reinforcement learning assigns a reward to a single sampled trajectory and then updates the policy. In both cases, the default unit is "one problem, one answer."

Exploratory modeling changes the training unit to a set of candidates:

• Input a problem x;
• Independently or controllably sample K candidates from the current policy;
• Score them using rules, test cases, reward models, or human preferences;
• Identify the highest-scoring answer in the candidate set;
• Update the model based on the set's performance.

If r(x, y) represents the score of answer y, the objective can be simplified as:

J_K(theta) = E[max(r(x, y_1), ..., r(x, y_K))]

y_i ~ pi_theta(. | x)

When K equals 1, this reduces to ordinary single-sample training. As K increases, the model gains room to explore. As long as one path succeeds, the entire candidate set receives a positive signal.

Its relationship with pass@k is straightforward. Assuming each sample is independent and the probability of success on a single sample is p, the ideal probability of succeeding at least once in K attempts is:

pass@k = 1 - (1 - p)^k

In practice, candidates are not independent. Models often repeat the same mistake using different wording, so actual pass@k will be lower than this ideal value. This is precisely why the value of exploratory modeling lies not only in increasing p, but also in reducing correlation among candidates: the model should genuinely try different approaches rather than generate K paraphrases.

It Addresses "Can It Do It?", Not Necessarily "Can It Do It on the First Try?"

The easiest misconception about this method is to equate improvements in best-of-K directly with improvements in model capability.

If a model's pass@1 remains unchanged and it merely becomes more likely to stumble upon the correct answer after 64 generations, then it is more like a better searcher than a more reliable one-shot answerer. This is still useful for mathematical research, code repair, and offline planning. But for low-latency chat, real-time completion, and token-billed API services, the gains may be offset by inference costs.

Therefore, at least three groups of metrics should be monitored together:

1. pass@1: Whether a single sample becomes more reliable;
2. pass@k: Whether the success rate continues to rise as the sampling budget increases;
3. Candidate diversity: Whether different answers cover different strategies, program structures, or proof paths.

If you look only at best-of-K, the model may learn to "cast a wide net." If you look only at pass@1, the model may converge too early on a frequent but brittle path. This is exactly the diversity-collapse problem seen in recent reinforcement learning for reasoning: training rewards keep rising, while model outputs become increasingly similar, until increasing K from 8 to 64 still fails to uncover any new solution.

The purpose of an exploratory objective is to explicitly tell the model that producing one genuinely different and effective solution within a set of answers is more valuable than copying the answer it currently considers most likely to succeed.

The Truly Difficult Part Is Credit Assignment

"Select the best candidate and train on it" can refer to at least three different approaches, and they should not be conflated.

Approach One: Best-Sample Distillation

The simplest implementation is to generate K candidates, select the highest-scoring one, and use it as supervised data for fine-tuning. This resembles online rejection-sampling fine-tuning: the model first creates data and then learns from its own high-quality samples.

The pseudocode can be written as:

for prompt in dataset:
    candidates = policy.generate(
        prompt,
        num_samples=K,
        temperature=temperature,
    )

    scores = verifier.score(prompt, candidates)
    winner = candidates[argmax(scores)]

    if max(scores) >= quality_threshold:
        loss = negative_log_likelihood(policy, prompt, winner)
        loss.backward()
        optimizer.step()

The advantages are stability, reproducibility, and the ability to directly reuse existing SFT infrastructure. The disadvantages are equally clear: the model sees only the winner and does not know why the other candidates failed. If the verifier favors a particular format, training will quickly amplify that formatting bias.

More importantly, this is not a strictly unbiased optimization of the set objective J_K, but rather an engineering-oriented form of distillation. It is often effective, but the presence of max in the formula does not mean that applying cross-entropy only to the winner is necessarily correct.

Approach Two: Reinforcement Learning With Set-Level Rewards

Another approach treats the K candidates as a whole and uses the set's highest score as the reward. The problem is that if the entire group receives a high reward because one answer is correct, should the other K-1 incorrect answers also be encouraged?

Assigning the same reward directly to every trajectory introduces substantial noise. A candidate may be completely wrong yet still receive a positive update simply because it happened to be grouped with a correct candidate.

A more reasonable approach is to estimate the marginal contribution of each trajectory: how much does the set's highest score decrease after removing candidate i?

credit_i = best_score(all_candidates)
           - best_score(candidates_without_i)

If removing an answer does not affect the set's best score, its marginal contribution is zero. If it is the only correct answer, it should receive most of the credit. This is more precise than winner-takes-all, but when a set contains multiple equally scoring correct answers, duplicate contributions and baseline estimation still need to be addressed.

Approach Three: Explicitly Encouraging Diverse Exploration

Simply increasing the temperature does not guarantee diversity. A higher temperature may only make the wording more random without enriching the reasoning strategies. Truly valuable exploration should occur at the semantic and decision-making levels: trying different theorems, different algorithms, different tool-call sequences, or different approaches to fixing errors.

The following constraints can be introduced in practice:

• Apply similarity penalties to duplicate candidates;
• Dynamically select strategies that have not yet been sufficiently explored for each candidate batch;
• Use upper confidence bounds to balance high-reward paths against novel paths;
• Let later candidates see summaries of earlier answers while explicitly requiring them to find different solutions;
• Measure diversity based on program execution traces, proof structures, or tool-call sequences rather than comparing text alone.

Exploration methods such as UCB-Con and Batch address precisely this tension between exploitation and exploration: the model cannot completely abandon paths with currently high success rates, but it also cannot allocate its entire sampling budget to the same type of answer simply because that answer is temporarily ahead.

The Verifier Determines the Method's Ceiling

Exploratory modeling has a very practical prerequisite: you must be able to determine which of the K candidates is better.

For coding tasks, unit tests and sandboxed execution can provide relatively reliable feedback. For mathematical problems, answer matching, symbolic computation, or proof checkers can be used. For agent tasks, the final environment state and tool-call results can be examined. These scenarios are well suited to exploratory training because their rewards are highly verifiable.

Open-ended writing, product recommendations, and complex factual question answering are much more difficult. A reward model may mistake linguistic fluency for factual correctness or greater length for more thorough analysis. When a model generates K candidates at a time, it gets more opportunities to "probe the verifier." What it finds may not be a high-quality answer, but rather a flaw in the verifier.

The larger K becomes, the more pronounced this Goodhart's law effect becomes: when a scoring metric becomes the optimization target, it increasingly ceases to be a reliable metric.

Therefore, at least three safeguards should be established when deploying this type of training pipeline:

• Verifier ensembles: Cross-check rules, execution results, and model-based evaluations to prevent a single judge from determining the entire reward;
• Holdout-set audits: Use tests and human-reviewed samples that are hidden during training to check whether the reward is being exploited;
• Out-of-distribution checks: Confirm that improvements are not limited to familiar templates or fixed problem types.

If the verifier's error rate is nontrivial, blindly increasing K can be dangerous. More candidates also give the model more opportunities to generate an answer that fools the judge.

A Larger K Is Not Always Better

Choosing K is first and foremost a computational tradeoff. During training, generating K trajectories for each problem causes inference costs to grow approximately linearly. If code sandboxes, theorem provers, or multiple reward models must also be run, verification may cost more than generation.

Meanwhile, marginal returns usually diminish. Increasing K from 1 to 4 may allow the model to quickly cover several major paths. Increasing K from 32 to 64 often yields only repetitive candidates. Rather than fixing a large K for every sample, a more sensible approach is to allocate the budget dynamically:

• Stop early when a simple problem passes verification with high confidence among the first few candidates;
• Continue sampling for difficult problems when candidates disagree substantially or their verification scores are close;
• Stop paying for duplicate answers once multiple independent correct solutions have appeared;
• Reduce the sampling budget for samples that consistently fail to produce positive rewards, or hand them off to a stronger teacher model.

This resembles early stopping and beam pruning in search systems. The training objective is to make every unit of computation produce new information, rather than mechanically filling a quota of K answers.

From best-of-K to Stable pass@1

Exploration capability ultimately needs to be compressed back into one-shot capability. Otherwise, performance can only be sustained through expensive test-time scaling.

A practical approach is to train in stages:

1. First, use an exploratory objective to expand solution coverage and establish sufficiently strong best-of-K performance;
2. Collect correct paths, failure reasons, and verification feedback from candidate sets;
3. Distill high-quality paths back into the base policy to improve pass@1;
4. Run exploratory training again to find solutions that the current policy still misses;
5. Evaluate across different values of K and different sampling temperatures to prevent the model from adapting only to a fixed budget.

This is very similar to evolutionary search: first generate a population of candidates, then gradually raise the best score through selection, repair, and rewriting. The difference is that exploratory modeling does not merely search at inference time. It also attempts to consolidate discoveries from that search into the parameters, giving the next round a stronger starting point.

In my view, this direction is highly valuable for verifiable reasoning tasks and better reflects the nature of the problem than simply pushing reinforcement learning rewards higher. For many difficult tasks, the model is not completely incapable; rather, the probability of the correct path under the current distribution is too low. K candidates expose latent capabilities, and training then turns low-probability capabilities into high-probability ones.

But it is not a free capability multiplier. Without a reliable verifier, Best-of-K degenerates into Best-of-Reward-Hacking. Without reasonable credit assignment, set-level rewards encourage large numbers of noncontributing answers. Without subsequent distillation, the model can maintain its performance only through higher inference costs.

How Developers Should Experiment With It

To validate this idea in an existing training stack, there is no need to begin with complex set-level reinforcement learning. A safer minimal experiment would be:

• Choose a code or mathematics dataset with deterministic feedback;
• Start with K=4 or K=8 rather than jumping directly to dozens of candidates;
• Save every trajectory, verification result, and measure of inter-candidate similarity;
• Begin with best-sample distillation, then compare it with single-sample SFT or ordinary RL;
• Report pass@1, pass@k, average generated tokens, and verification costs together;
• Separately check whether incorrect answers are becoming increasingly homogeneous.

The curve most worth watching is not the training reward, but the shape of pass@k as k increases. If the curve quickly flattens as k grows, the model is not truly exploring. If pass@k rises while pass@1 remains unchanged for a long time, the search capability has not yet been successfully distilled. Only when both improve can we say that training has preserved diversity while also improving the quality of the default policy.

Ultimately, exploratory modeling reminds developers of something that single-answer training can easily obscure: a model must learn not only how to produce an answer, but also how to allocate attempts under uncertainty. For the next generation of reasoning models, this search strategy may be as important as parameter count and data volume.

References

• A Survey of Reinforcement Learning in Large Reasoning Models (Part 1): An overview of reinforcement learning for reasoning, set-level objectives, credit assignment, and methods for preserving a policy's ability to explore.
• Hugging Face TRL: A widely used post-training library for large models that can be used to build supervised fine-tuning, reward modeling, and reinforcement learning experiments.
• vLLM: A high-throughput model inference and batch-sampling framework suitable for implementing multi-candidate rollouts during training.