Module cpu 

CPU f32 reference forward for qwen4-exp (decode form, one token at a time). Slow by design: it is the oracle for the Metal path and mirrors the transformers implementation (modeling_qwen4_exp.py) step for step.

Conventions verified on the checkpoint: RMSNorm weights are MLX-sanitized (plain w * norm(x), no 1 + w); the DeltaNet output gate is a sigmoid (output_gate_type); attention uses a per-head sigmoid output gate.

Structs

AttnWeights

CpuModel

DeltaState

DeltaWeights

HcWeights

KvCache

Layer

MoeWeights

MtpWeights

The one-layer MTP draft head: folds the next token’s embedding into the trunk’s wide residual, runs one attention + MoE block, collapses with its own mixer and reuses the trunk’s LM head.

PleState

PleWeights

State

Enums

Mixer

Functions

bf16_dot 🔒

bf16_matvec 🔒

causal_conv 🔒

Causal depthwise convolution over packed [q | k | v] channels. History is oldest-first; preserve accumulation order for the CPU oracle.

grouped_rms_norm 🔒

RMSNorm applied independently to each group-wide slice, with a full width weight vector.

hc 🔒

inject 🔒

load_layer 🔒

mlp 🔒

rms_norm_shift 🔒

RMSNorm with the raw HF weight convention: x * inv * (1 + w).

rope_partial 🔒

Partial RoPE over the first rot dims with half-split pairing (rotate_half): pair (j, j + rot/2) uses inv_freq theta^(-2j/rot).