GLM 5.3, Sparse Attention and the Cost of Inference

Jordan Nanos welcomes SemiAnalysis colleagues Kimbo Chen and Wenyao Gao to discuss GLM 5.3, from sparse attention and inference benchmarks to post-training, cybersecurity and hardware support.

Chen explains that sparse attention selects a subset of relevant tokens while retaining the core attention operation. Linear attention takes a different approach by changing the fundamental algorithm. The goal is to make long contexts more tractable.

Since GLM 5, Z.ai has used DeepSeek Sparse Attention, which combines a lightning indexer for token selection with sparse attention based on multi-head latent attention. Other labs have continued experimenting with alternative attention designs.

Chen says DeepSeek’s later compressed and heavily compressed attention variants build on these ideas with additional compression. Nanos points to the cost benefit of serving long sequences more efficiently, while noting debate over whether hardware restrictions contribute to Chinese labs’ emphasis on efficiency.

Gao describes a serving workload with many concurrent users, each potentially carrying a long context. GPU memory is finite, and increasing concurrency requires more space for the cached information used during inference.

HiCache reuses prefix caches and can offload data from high-bandwidth GPU memory to host DRAM. Gao says increased concurrency changes how much host memory is used; differences in per-chip memory also affect results.

Chen describes HiSparse as a sparse-attention counterpart to HiCache. It moves cache data to a larger, lower-bandwidth memory tier and brings relevant portions back for computation.

He stresses a distinction: sparse attention reduces the amount of data read for an attention operation and helps with bandwidth, but does not by itself eliminate the need to retain the full context. Cache capacity remains a concern because tokens must still be available for selection.

Nanos points to the benchmark’s modeled throughput per dollar. Gao says the model’s computational efficiency should not be confused with eliminating memory demands; offloading between HBM and DRAM remains an important serving technique.

Gao says NVIDIA initially led the measured configurations, but AMD’s cost at approximately 150 tokens per second per user later approached that of NVIDIA B200 configurations. The comparison depends on the particular benchmark, software and hardware configuration.

The InferenceX dashboard can replay how performance changed after the model’s early August release. Nanos notes that AMD MI355 results improved substantially, including at interactive speeds around 100 tokens per second.

Gao attributes part of the improvement to continuing work on AMD’s ATOM inference engine. Hardware capabilities are fixed for a given chip, but software can be repeatedly optimized, changing the cost and throughput achieved on it.

Nanos asks about host DRAM, GPU memory capacity and bandwidth. He notes that some investors feared efficient models would reduce memory demand; more efficient inference can instead enable greater concurrency and continued demand for memory.

Gao says offloading can support more users and increase aggregate output, but introduces latency, especially in the long tail. The appropriate balance depends on the workload and the price a user will pay.

As an example of a different trade-off, Gao describes TileRT’s decode mega-kernel, which reduces launch overhead and delivers up to 470 tokens per second in a small-batch case. Its low latency comes with limitations on serving large batches.

Both guests say they do not generally use ultra-fast coding modes: Gao finds the price high and the difference hard to feel for her ordinary agent tasks. A time-sensitive trading firm, she suggests, might value a fast response differently.

Chen argues that bandwidth becomes especially important at very low latency, where performance is constrained by how quickly model weights and relevant data can be loaded for relatively little computation.

Capacity still matters, but cache hierarchies and overlapping computation with transfers can help manage its limits. Chen interprets the discussed Rubin memory trade-off as a bet on retaining bandwidth while using networking and other methods to address capacity.

Asked whether Z.ai will retain its attention architecture, Chen speculates that the GLM 5 series may continue with it, while future models could explore alternatives. He points to GLM 5.3 Flash as an example combining Kimi Delta Attention and DeepSeek Sparse Attention.

The discussion turns to capabilities and post-training. Nanos says he uses GLM 5.3 as a fallback for some authorized cybersecurity work when other models refuse; he considers it capable for coding and security tasks. This is his experience, not a general benchmark.

The speakers discuss comparisons with leading proprietary models. Chen cautions that GLM 5.3 is not equivalent to the highest class they mention, although he regards its progress as notable given that the improvement came through post-training.

Chen says Z.ai has concentrated on improving the GLM 5 architecture through post-training while other labs have more frequently changed architectures. He describes familiar reinforcement-learning methods, including GRPO, as part of the general picture.

He says an initial synchronous GRPO stage favors training stability. Longer agent trajectories can dilute reward signals, so later stages make different algorithmic and systems trade-offs, including how to handle the staleness of policies used to generate data.

Nanos asks whether these techniques are fundamentally unique. Chen says much of the discussion so far is not unique to Z.ai. Gao emphasizes that the model’s benchmark improvements through post-training alone show what a team might do without pretraining from scratch.

The guests say the improvement from GLM 5.2 to 5.3, including on the security tasks they discuss, was striking. Chen cautions that Z.ai has not published a detailed technical report since GLM 5, so some details come from public statements and researcher conversations.

Chen highlights single rollout asynchronous optimization, or SAO, which he describes as using a critic alongside the main model to evaluate rollouts. He also points to the open-source Slime post-training framework as an important infrastructure contribution.

Nanos suggests that efficient infrastructure is an advantage in its own right: with the same compute budget, teams may be able to run more experiments or jobs. Chen agrees and says Z.ai has discussed using its models to help improve system efficiency.

The examples include bringing up and verifying kernels on different hardware and monitoring system-level workloads. Chen says both the algorithmic work and the open infrastructure may contribute to faster research and training.

Nanos reads from the GLM 5 paper: the authors say they adapted the software stack to seven Chinese chip platforms, including Huawei Ascend, Moore Threads, Hygon, Cambricon, Kunlunxin, MetaX and Enflame.

Chen says Z.ai also publicly claimed its GLM 5.3 Flash model was served on Chinese hardware under the name aux alpha. He treats the claim as evidence of work across kernels and serving systems, while acknowledging that he is relying on the company’s account.

The timing of GLM 5.3 Flash prompts Chen to speculate about a short development cycle. He makes clear that he does not know exactly when the work began, so the proposed timeline is an estimate.

Chen describes possible hardware and software co-design: attention configurations can be chosen with the target hardware’s arithmetic characteristics in mind. He gives DeepSeek’s H800-oriented design as a comparison.

His rough calculation for GLM 5 suggests an arithmetic intensity near 120 that he says resembles Moore Threads hardware. He explicitly says this does not prove the model was optimized for, or served on, that particular hardware; he has no confirmation.

Gao says better models and suitable tools can shorten research and development cycles, including GPU kernel development. The speakers wonder how much AI assistance contributed to recent high-performance kernels but do not know the answer.

Looking ahead, Chen speculates that the mixed attention architecture in GLM 5.3 Flash could be scaled up. Gao expects further work both on model capability and inference efficiency, particularly for organizations sensitive to serving costs.

Gao points to companies adapting open models for specialized applications. She is interested in whether post-training can improve useful capabilities at a fixed model size as well as what future GLM releases bring.

Chen adds that human effort and data quality are important but insufficiently explained parts of post-training. He suspects labeling plays a role, while acknowledging that the exact work behind the data is not public.

Gao is also interested in smaller efficient models that are good enough for specific tasks at low cost. Nanos closes with the team’s view that efficiency gains may encourage more token use and continuing demand for memory and bandwidth.