Huawei Open-Sources 505B-Parameter Pangu Pro: Total Parameter Count Is No Longer the Only Metric for Large Models

Today (July 31), Huawei officially open-sourced openPangu-2.0-Pro, releasing the model weights, basic inference code, and technical report simultaneously. It is a mixture-of-experts model with approximately 505B total parameters and around 18B parameters activated per token. It supports a 512K context window and was trained on approximately 34T tokens of data.

This release fulfills the timeline Huawei announced at HDC 2026. On June 30, the smaller openPangu-2.0-Flash was released first; one month later, the higher-end Pro version arrived on schedule.

Judging by the numbers alone, 505B could easily be seen as yet another round in the parameter-count race. But what makes openPangu-2.0-Pro truly noteworthy is not how many parameters Huawei has packed into it. Rather, it is Huawei's attempt to answer a more practical question: Given the available domestic computing infrastructure, how can an extremely large model be made fast and efficient enough to enter production?

505B Represents Model Capacity; 18B Is Closer to the Cost of a Single Inference Step

openPangu-2.0-Pro uses an MoE, or mixture-of-experts, architecture. It has approximately 505 billion total parameters, but only around 18 billion of them are used to process each token.

It can be understood as a large consulting firm with hundreds of experts. The firm's collective knowledge depends on all of its experts, but each individual project does not require everyone to participate at once—only a small group of the most relevant experts needs to be assigned. The 505B figure determines the expert capacity the model can accommodate, while 18B is closer to the amount actually involved in a single computation.

The value of this design is straightforward: the model can continue expanding its knowledge capacity and capability ceiling without inference costs increasing linearly with the total parameter count. Activating 18B parameters means that, in theory, it can generate outputs using far less computation than a dense model of the same total size. However, this does not mean deployment only needs to accommodate 18B parameters. The complete weights still need to be stored, loaded, and distributed across devices, while expert routing also introduces communication overhead.

The 505B/18B configuration therefore favors inference efficiency, but it cannot simply be interpreted as "getting the capabilities of a 505B model at the cost of an 18B model." Actual throughput and latency also depend on expert parallelism, device interconnects, quantization strategies, batch size, and how well the inference framework is optimized for Ascend hardware.

This is also one of the biggest differences between openPangu and conventional open-source models: rather than being trained first within a general-purpose GPU ecosystem and only later adapted to Ascend, it was designed around Ascend NPUs from the beginning of training. For enterprises that have already deployed Huawei computing infrastructure, this native compatibility has more practical significance than a few extra points on a benchmark leaderboard.

A 512K Context Window Does Not Mean All 512K Tokens Can Be Used Effectively

openPangu-2.0-Pro supports a 512K context window, enough to hold the key files from a large code repository, hundreds of pages of technical documentation, or a set of Agent operation logs spanning several hours.

Long context windows are no longer a rare selling point. The challenge has shifted from "Can the information fit?" to three more specific questions:

• Can the model accurately retrieve information buried deep in the context?
• How much do time to first token and generation throughput deteriorate as the sequence grows?
• Can KV cache usage be kept within an acceptable range?

Huawei's solution combines MLA, DSA, and SWA. MLA compresses attention-related cache overhead. SWA, or sliding-window attention, focuses on local relationships between neighboring tokens. DSA handles sparse global aggregation across long distances. The model mixes DSA and SWA layers at a 1:2 ratio.

This is somewhat like reading an extremely long technical manual: most of the time, you only need to look at content near the current chapter, rather than compare every page individually with every other page. A global index is only needed when encountering term references or cross-chapter dependencies. SWA is responsible for "looking nearby," while DSA "searches the entire book."

If standard global attention were used throughout, computation and memory-access pressure would grow increasingly quickly as the sequence length increased. Layering sparse global attention with local windows can eliminate a great deal of unnecessary pairwise computation. At a length of 512K, this is not merely a nice-to-have optimization—it is a prerequisite for making the model practical to run.

However, the officially stated "512K support" is currently, first and foremost, an architectural and engineering specification. It does not directly prove that the model can maintain stable comprehension across the full 512K range. Developers should wait for tests such as needle-in-a-haystack retrieval, multi-hop retrieval, long-range code dependency analysis, and real-world document question answering, rather than treating the context window size as equivalent to effective memory length.

Four-Stream Residuals and Three-Head MTP: Huawei Is Extracting More From Every Inference Step

In addition to its attention architecture, openPangu-2.0-Pro upgrades conventional residual connections to a four-stream mHC architecture. Traditional Transformers generally pass information layer by layer along a single primary residual path. mHC instead allows multiple hidden-state streams to evolve and interact in parallel, with the aim of improving representational diversity and generalization.

The potential benefit of this structural change is that it may reduce interference caused by different types of information being forced through the same channel in deep networks. However, it also increases the complexity of training stability, operator implementation, and inference scheduling. Whether the trade-off is ultimately worthwhile still needs to be assessed through ablation studies: if the capability gains are small while deployment becomes significantly harder, it may amount to little more than an elegant architectural design. If it substantially improves performance at the same number of activated parameters, however, it could become a key differentiator for openPangu 2.0.

The more intuitive inference-side change is three-head MTP. A conventional autoregressive model predicts one token at each step, while MTP additionally attempts to predict multiple subsequent tokens. openPangu-2.0-Pro predicts three additional tokens at once, after which the main model verifies the candidates.

It is similar to having an assistant draft the next three sentences in advance for the lead writer to review quickly. When the predictions are correct, the model can accept multiple tokens at once, reducing the number of sequential decoding steps. When the predictions are inaccurate, it falls back to the conventional generation process. The actual speedup depends on the acceptance rate of the candidate tokens, so the gains are generally more pronounced for highly predictable tasks such as code completion and structured output than for open-ended creative writing.

Huawei also uses the Muon optimizer during training, with the aim of accelerating convergence for large-scale models. Given that openPangu-2.0-Pro was trained on approximately 34T tokens, even a percentage-point improvement in training efficiency translates into substantial savings in compute and time.

Taken together, these designs make openPangu-2.0-Pro's direction clear: instead of relying solely on parameter scaling to improve capabilities, it reduces costs simultaneously across four areas—attention, residual topology, training optimization, and decoding.

Post-Training Unifies Fast and Slow Reasoning With Agent Scenarios in Mind

During post-training, openPangu-2.0-Pro underwent unified fast-and-slow fine-tuning and multiple domain-specific reinforcement learning stages, with different capabilities consolidated into a single model through online distillation, or OPD.

"Unifying fast and slow reasoning" can be understood as allowing the model to retain two operating modes: responding quickly to simple questions and investing more reasoning steps in complex problems. For Agents, this is more sensible than applying the same compute budget to every request. Looking up a configuration item and planning a cross-system migration clearly should not incur the same reasoning cost.

Multiple domain-specific reinforcement learning suggests that the model may have been optimized separately for different capabilities, such as mathematics, coding, tool use, or instruction following, before those specialized capabilities were integrated into a unified model through online distillation. The significance of online distillation is not limited to having a large model teach a smaller one. It can also enable different policies and capabilities within the same training process to transfer knowledge to one another, reducing conflicts between specialized optimizations.

However, Huawei has not yet provided sufficiently comprehensive third-party comparative evaluations in its open-source announcement. Determining whether the model truly achieves "speed without sacrificing strength" will require examining at least four categories of metrics:

1. Whether its general knowledge, mathematics, and coding capabilities outperform models with a similar number of activated parameters;
2. Recall and reasoning stability under a 512K context window;
3. Time to first token, output speed, and concurrent throughput when deployed on single- and multi-card Ascend configurations;
4. Success rate, cost, and error-recovery capabilities during continuous Agent tool use.

Until sufficient data in these areas is made public, the most prudent assessment of openPangu-2.0-Pro is: its architectural direction has clear engineering value, but the figures 505B and 512K alone are not enough to prove that it has entered the top tier of open-source models.

Open Weights and Inference Code Are the Starting Point, Not the Finish Line

Huawei has released the model weights, basic inference code, and technical report. This is already important for developers: at a minimum, they can inspect the model architecture, evaluate deployment options, and validate inference in an Ascend environment.

But for a complete open-source ecosystem, model weights are only the first layer. Pre-training code, post-training code, training operators, data-processing pipelines, evaluation suites, and deployment tools determine whether the community can truly reproduce, modify, and extend the model.

At HDC 2026, Huawei committed to gradually open-sourcing seven major components beginning on June 30, including previously unreleased pre-training code, post-training code, and training operators. The current schedule is:

• On June 30, openPangu-2.0-Flash, with 92B total parameters and 6B activated parameters, was released first;
• On July 31, openPangu-2.0-Pro, with 505B total parameters and 18B activated parameters, released its weights and basic inference code;
• The remaining training and supporting components are scheduled to be released throughout the second half of 2026.

It is therefore more accurate to say that the open-source pieces of openPangu 2.0 are still being assembled, rather than that the process is complete. For users outside the Ascend ecosystem in particular, how easily the model can be ported to other hardware, whether community inference frameworks add support, and whether the license permits commercial use will all affect how far it can ultimately go.

openPangu Is Really Competing to Become the Gateway to the Ascend Ecosystem

The significance of openPangu-2.0-Pro extends beyond Huawei releasing another large model. It is more like a large-scale reference implementation for the Ascend software stack, showing developers how to train MoE models on Ascend, handle extremely long contexts, implement multi-token prediction, and organize large-scale inference.

Richard Yu previously explained that much of Huawei's computing capacity also needs to support demand from other domestic enterprises, leaving limited resources for its own use. As a result, rather than simply expanding model size, the team is placing greater emphasis on latency and throughput. This also explains why the Pro version uses a 505B-total/18B-active design instead of continuing to pursue an even larger dense parameter count.

This is a pragmatic choice. Enterprise customers today do not lack models with "lots of parameters." What they lack are models that can be deployed reliably on existing clusters, with predictable costs and controllable latency. In on-premises scenarios—particularly in government, enterprise, manufacturing, and finance—hardware compatibility and operational maintainability are often more important than public benchmark rankings.

Its limitations are equally clear. Deep optimization centered on Ascend can improve the model's efficiency on Huawei hardware, but it may weaken its cross-platform appeal. The CUDA ecosystem offers mature tools for quantization, inference, fine-tuning, and monitoring. Developers considering a move to another hardware and software stack will need to see sufficiently clear performance or supply-chain benefits.

The next phase of competition for openPangu-2.0-Pro will therefore take place not only on model benchmark tables, but also in operator coverage, framework compatibility, documentation quality, deployment barriers, and community responsiveness. If all seven major components are fully released, accompanied by reproducible throughput results and real-world industry use cases, it could become a highly influential foundation model within the Ascend ecosystem. If the community can only download the weights but cannot run or further train the model cost-effectively, then its 505B scale will instead become a barrier to adoption.

An Engineering-Focused Large Model Worth Validating

openPangu-2.0-Pro does not follow the simplistic path of treating the largest parameter count as synonymous with the best model. Its 18B-active MoE, hybrid DSA/SWA attention, four-stream mHC architecture, three-head MTP, and Muon optimizer all serve the same objective: making an extremely large model run more efficiently on limited computing resources.

From an engineering perspective, it is more compelling than simply announcing a 505B model. From the standpoint of ecosystem maturity, however, it remains at a stage where real-world testing and further tooling are still needed.

For teams already using Ascend hardware, this open-source release is worth validating as soon as possible, with particular attention to long-context performance, concurrent throughput, and per-token cost in real business workloads. Other developers need not rush to migrate their technology stacks based solely on the 505B and 512K figures. It would be more prudent to first assess the forthcoming training components, cross-framework support, and third-party benchmarks.

Huawei has put the model weights and basic inference code on the table. What determines openPangu-2.0-Pro's impact from here will no longer be the parameters presented at a launch event, but whether developers can actually get it running.

References

• ITHome: Huawei's Pangu openPangu-2.0-Pro Model and Technical Report Released as Open Source — Covers the July 31 open-source release, model scale, training data, and key architectural details.
• Zhihu: Richard Yu Declares "There Is No Number Two" as Huawei Pangu 2.0 Arrives — Provides additional information on the positioning of the openPangu 2.0 series, the specifications of the Pro and Flash models, and the Ascend ecosystem context.