Sun Yutao Reads the Kimi K3 Report: Architecture, Training and Infrastructure

Zhang Xiaojun invites Sun Yutao to read Moonshot AI’s Kimi K3 technical report, connecting model architecture, pretraining, post-training and infrastructure to related research.

Sun describes his research background and interest in large-language-model architecture, linear attention and training efficiency.

He reviews early linear-attention work: researchers wanted to improve long-sequence computation costs while maintaining performance on general tasks.

They discuss mixing attention mechanisms. Theoretical complexity, actual hardware utilization and model quality must be considered together; the mixture ratio is not the only metric.

He introduces the idea of reusing model layers in a loop: parameters can be reused, but computation and inference cache costs do not automatically disappear.

Before turning to K3, Sun outlines his approach: start with the model design, connect it to earlier papers, then discuss training and deployment.

The K3 report describes a mixture-of-experts model with different total and active parameter counts and considers capacity, context and system efficiency together. It reports about 2.8 trillion total parameters and 104 billion activated per token.

He reviews work including RetNet, Gated DeltaNet and Kimi Linear, explaining how linear attention uses state updates and decay to process sequences.

Discussing finer-grained decay, he points to a tradeoff between expressiveness and kernel implementation complexity. A mathematical improvement must also execute effectively on hardware.

He explains the joint design of numerical precision and tiled computation: controlling decay ranges can help avoid instability in finite-precision arithmetic.

The discussion turns to multi-head latent attention and gating. Sun compares cache, performance and conditions for different designs, saying the right choice depends on the complete system.

He reviews the history of residual connections and layer normalization, explaining why training stability becomes important as networks grow deeper.

On Attention Residuals, he discusses allowing layers to selectively aggregate earlier representations and connects the idea to earlier dense-connection methods.

Discussing latent MoE, he explains that computation in a latent expert space can reduce communication and weight-access costs, depending on the configuration and implementation.

He introduces activation functions and numerical-range control, comparing hard clipping with smoother bounds and emphasizing both model and operator design for stable training.

He discusses the Muon optimizer, activation outliers and normalization or bounding strategies at different locations. Approaches differ in cost and applicability.

On expert routing and load balancing, he introduces Quantile Balancing and discusses avoiding overload of particular experts or devices.

They return to combinations of sparse, full and linear attention. Sun stresses evaluating algorithmic properties together with kernels, caches and hardware costs.

Zhang asks whether K3 mainly combines prior work. Sun argues that progress often accumulates across many efforts and that integration and engineering also require innovation.

The pretraining section covers scaling and training budgets. Token counts, learning-rate schedules and continued-training strategies can affect results.

For long-context extension, he discusses combining RoPE with no positional encoding and notes that outcomes depend on how other layers represent position.

Turning to post-training, they discuss supervised fine-tuning, reinforcement learning and signals for different capabilities. The report describes one implementation, not a mandatory sequence for all models.

Sun explains multi-teacher on-policy distillation: a student receives teacher signals on its own generated samples, with different teachers contributing capabilities. Teacher selection and sample quality remain important.

He discusses deployment-aware post-training and draft models, aligning training with inference conditions and exploring speculative-decoding efficiency.

The infrastructure section returns to linear-attention kernels: chunking, parallelism and state transfer affect efficiency while requiring numerical correctness.

For distributed training, he explains expert-parallel load balancing and how planning expert placement and redundancy can reduce uneven work across devices.

He discusses overlapping communication with computation: when model and system design reduce communication costs, different methods can hide transfer latency.

Long-context and multimodal training can create uneven loads across devices. Teams need to coordinate parallelism, vision encoders and memory use.

He introduces reinforcement-learning infrastructure including memory reuse and sandboxes. Engineering must account for isolation, stability and the cost of large-scale operation.

The inference section covers caches, kernels and serving schedules. A faster individual operator does not necessarily mean a faster system; the full workload must be measured.

Sun closes by emphasizing that model and training-system progress usually comes from sustained research, teamwork and accumulated work across the field, not merely increasing a few size parameters.