Module gpu 

Metal decode path for qwen4-exp: row-batched steps (prefill chunks and speculative verification of MTP drafts) over an explicit wired expert pool.

Dense weights are one zero-copy buffer over dense.bin. Routed experts live in a fixed pool of page-aligned slots (Metal wires whole buffers per command buffer), LRU managed here and filled by uncached parallel reads. One command buffer per step: after each layer’s router the GPU signals a shared event and waits for the CPU to publish that layer’s expert slots; a one-layer lookahead router lets most fetches run in the background.

A step runs nb consecutive positions at once (activations are row-major [nb][...]). The gated DeltaNet scan snapshots its state after each row so a partially accepted verify batch can roll back.

Modules

activity 🔒

Cumulative routed-expert activity. Verification and draft rows count even when later rolled back. IO counters describe requested reads, not completion.

attention 🔒

QSA indexing, q8 KV append, and selected decode attention.

budget 🔒

Expert-pool sizing after fixed buffers and host reservations.

decode 🔒

Trunk row-batch execution and single-token stepping.

deltanet 🔒

Causal convolution and recurrent DeltaNet decode dispatch.

dispatch 🔒

Buffer binding, dispatch geometry, dense projections, and transfers.

experts 🔒

Router and resident/fetched expert compute dispatch.

hyperconnection 🔒

Grouped normalization and gated residual mixing/injection.

load 🔒

Model loading, pipeline creation, and fixed/pool allocation.

memory 🔒

Runtime resource counters shared by CLI telemetry and control status.

mtp 🔒

Draft-head execution and folded/chained MTP inputs.

params 🔒

Host/shader ABI: preserve field order and match the Metal declarations.

phases 🔒

Timestamp the existing compute passes; resolve only after their normal wait.

ple 🔒

N-gram prefetch/gather and PLE gating/convolution dispatch.

prefill

Prefill engine: a prompt chunk of up to a few thousand tokens runs through the model layer by layer, so each layer’s experts are streamed once for the whole chunk instead of once per few tokens.

residency 🔒

The expert pool: which records the GPU may touch, LRU managed here, with two backings behind one interface.

sampling 🔒

Final hyper-connection read, logits projection, and greedy argmax.

state 🔒

Commit/rollback, sequence reset, and prefix checkpoints.

streaming 🔒

Decode expert streaming: GPU routing, CPU slot publication, and file IO.

Structs

Attn 🔒

Delta 🔒

ExpertActivity

ExpertCounters

GLayer 🔒

Gpu

HalfSet 🔒

Half even/odd streams plus per-32 group sums of a row batch, the input format of the multi-row Q4 matvecs.

Hc 🔒

HcBufs 🔒

Buffers of one gated-residual read (the main set, or the lookahead’s).

LaEntry

One lookahead prediction and what became of it (CHERENKOV_DUMP_LA).

LayerStats

MemoryStats

Capacities and observations, not a partition of process memory. CPU prefix checkpoints and session history are reported separately by the server.

Moe 🔒

Mtp 🔒

The MTP draft head (see cpu.rs MtpWeights).

PhaseStats

Pipes 🔒

Ple 🔒

PredictionStats

PrefixState 🔒

Compact sequence checkpoint; no weights, expert slots, or device event counters.

Q 🔒

Affine-Q4 projection: dimensions and byte offsets in its bound weight buffer.

QuantStats

ReadStats

Scratch 🔒

T 🔒

Byte offset of a bf16 tensor in dense.bin.

Enums

GpuTiming

Mix 🔒

Constants

ARGMAX_TGS 🔒

ATTN_MAX_WG 🔒

ATTN_TB 🔒

IDS_IN 🔒

ids buffer layout: trunk inputs, trunk argmax, MTP inputs, MTP argmax.

IDS_MTP_IN 🔒

IDS_MTP_OUT 🔒

IDS_OUT 🔒

MAX_NB

Rows per step (prefill chunk, or 1 + drafts when verifying).

MAX_SNAP

Rollback planes: verify batches may have up to this many draft rows.

SLOT_STRIDE 🔒

Slots per layer row of the expert slot table (last entry = union size).

Functions

kv_q8_side 🔒

layer_cap 🔒

Debug: CHERENKOV_LAYERS=N runs only the first N decoder blocks (for bisecting one execution path against another).

set_bytes 🔒

union_of 🔒

Unique entries in first-seen order.

Type Aliases

Buf 🔒

Enc 🔒

Pso 🔒

RouteStep

Tokens, expert IDs, routing weights, and misses for one step. Keep the tuple layout for compatibility with existing research dumps.