Expand description
Standalone qwen4-exp text inference on Apple Silicon.
Modules§
- config
- Server configuration: built-ins, one TOML file, then explicit CLI overrides.
- control
- Local control: one bounded JSON exchange per connection, authenticated by UID.
- download
- Pinned Hugging Face downloads. The Hub client handles retries, locking and Xet transfers.
- kernels 🔒
- Metal translation units shared by the engine and GPU tests.
- metal 🔒
- model
- Hardware-independent checkpoint inspection and preparation requirements.
- nn 🔒
- Small CPU math helpers for the f32 reference path.
- options
- prefix_
cache 🔒 - Bounded LRU checkpoints of complete hybrid-model sequence state.
- prompt 🔒
- Checkpoint-owned Jinja formatting, compiled once when the tokenizer loads.
- quant 🔒
- MLX affine quantization, CPU side. Layout (bits=4, group_size=64):
- qwen4_
exp - qwen4-exp (model_type qwen4_exp_text): 48-layer hybrid of Gated DeltaNet and sparse attention, 512-expert MoE per layer, four-stream gated residual (“hyper connections”), a hashed n-gram embedding injected at one layer, and a one-layer MTP head. 125B parameters, 6B active.
- runner
- CLI generation, prompt processing and shared decode entry points.
- sampling
- Request-owned sampling state. GPU logits are borrowed after synchronization.
- server
- Server startup and shared job ownership. One worker owns the loaded GPU.
- storage
- XDG locations and the durable model layout. Resolving paths never creates files.
- tensors 🔒
- tok 🔒
- units 🔒
- Byte units used by memory limits, storage sizes and progress reporting.