Module qwen4_exp 

qwen4-exp (model_type qwen4_exp_text): 48-layer hybrid of Gated DeltaNet and sparse attention, 512-expert MoE per layer, four-stream gated residual (“hyper connections”), a hashed n-gram embedding injected at one layer, and a one-layer MTP head. 125B parameters, 6B active.

Weights come from the packed layout written by pack. Metal wires whole buffers, so every expert record must be its own page-aligned buffer, and the MLX shards are not even 4-byte aligned on disk.

Modules

config 🔒

Model configuration for the qwen4_exp text engine.

cpu

CPU f32 reference forward for qwen4-exp (decode form, one token at a time). Slow by design: it is the oracle for the Metal path and mirrors the transformers implementation (modeling_qwen4_exp.py) step for step.

gpu

Metal decode path for qwen4-exp: row-batched steps (prefill chunks and speculative verification of MTP drafts) over an explicit wired expert pool.

lowbit

Building the low-bit expert stores from the 4-bit one, in process. experts2.bin and experts3.bin coexist beside the source store; choosing or rebuilding one precision leaves the other file intact.

manifest 🔒

On-disk packed tensor layouts, independent of runtime sequence state.

pack

Pack qwen4-exp checkpoints into aligned files. MLX Q4 tensors are copied bit for bit; native BF16 tensors are converted as the files are written.

packed

Zero-copy views over the packed qwen4-exp layout (pack.rs).

Structs

DenseEntry

Where a dense tensor lives inside dense.bin.

ExpertLayout

Expert record layout inside experts.bin. One record per (layer, expert): three packed 4-bit weight matrices followed by their scales and biases, padded to a page multiple so each record can be its own Metal buffer.

Manifest

NgramLayout

N-gram table layout inside ngram.bin: one row_bytes record per hashed id holding the packed 4-bit row, then its scales, then its biases.

Qwen4ExpConfig

Text-model configuration, parsed from config.json’s text_config.

RopeParams

Constants

PAGE