Cherenkov

Cherenkov is a Rust and Metal inference engine for experimental Qwen4 MoE models (qwen4_exp) on memory-constrained Apple devices. It runs from the 4-bit quantized checkpoint, streaming experts from SSD into a bounded GPU cache. The saved benchmark ran on a 32 GB M4 MacBook Air and used about 21 GB of Metal allocations.

Get started

Cherenkov runs on Apple Silicon and macOS. To build from source, install the Xcode command-line tools and Mise.

mise install
mise exec -- cargo build --release
target/release/cherenkov prepare hf://Sawfwair/Qwen3.8-Flash-Next-MLX-4bit@6cc9bbc0 --name flash
target/release/cherenkov serve --model flash

Cherenkov accepts native BF16 checkpoints or MLX affine 4-bit weights quantized in groups of 64 (32 for the 160-column n-gram table). We tested Sawfwair/Qwen3.8-Flash-Next-MLX-4bit at revision 6cc9bbc0, used in the commands above.

Important

The mlx-community conversion uses groups of 32 for the main weights, so Cherenkov cannot load it.

The built-in packer converts BF16 weights to 4-bit as it writes aligned records. It preserves existing MLX quantized weights bit for bit. No Python or MLX runtime is required. prepare registers the source, downloads it, and prepares the model. Allow roughly 210 GB during preparation. Packing removes the temporary source after success; add --keep-source to retain it. See storage and downloads for paths and HF_TOKEN.

For a checkpoint already on disk, register its parent store once:

target/release/cherenkov store add models /path/to/models
target/release/cherenkov prepare disk://models/Sawfwair/Qwen3.8-Flash-Next-MLX-4bit --name flash
target/release/cherenkov serve --model flash

This example uses /path/to/models/Sawfwair/Qwen3.8-Flash-Next-MLX-4bit. A direct checkpoint path also works. The alias is optional; commands accept the source URI too. See the model index for HF cache stores.

Benchmarks

The benchmark ran on Apple M4 hardware with 32 GiB of memory. The engine reported 20.98 GB of Metal allocations. The report contains 80 valid samples from revision 93c514f.

The inference rates exclude loading and store construction. Answer lengths vary, so compare completion times in the full report.

Full report.

Settings with a deadline cut are not reproducible.

All timings and outputs.

Pelicans

These are unedited model outputs from the benchmark.

4-bit	4-bit / 2-bit misses + cut

3-bit	2-bit

Run the full suite, save its answers and pelicans, and refresh this section:

cargo xtask bench flash --build-stores --update-readme
# Regenerate from a completed run without inference:
cargo xtask readme results/baseline-2026-09-09

All outputs live in results/. See the benchmark method, paired prefill measurements, and current validation.

Run

Server: serve keeps the model loaded and caches repeated prompt prefixes. Connect an OpenAI-compatible client to http://127.0.0.1:8080/v1 with model cherenkov. Chat and text completions support streaming and sampling. The server interleaves up to two active requests by default, with cancellation and optional retained sessions. Chat renders the checkpoint’s Jinja template with thinking disabled. Reasoning-effort controls are not exposed yet. See the HTTP API.

CLI: pass an indexed model or local directory and a prompt to generate directly.

target/release/cherenkov flash \
  'Explain hash collisions.' --max-tokens 256
target/release/cherenkov flash \
  'Explain hash collisions.' --experts 3
target/release/cherenkov status
target/release/cherenkov --help

List registered models and inspect their stores:

target/release/cherenkov model list
target/release/cherenkov inspect flash

model remove <reference> releases the registration. model gc --dry-run previews unused managed stores; model gc deletes them. See the model index for local sources, exports, and retention.

Statistics

Query the running server from another terminal:

target/release/cherenkov dash
target/release/cherenkov stats summary
target/release/cherenkov stats layers
target/release/cherenkov stats experts 7
target/release/cherenkov stats layers --json

stats shows formatted summaries; --json returns all fields. In dash, select a row for details. Press Enter on a layer to inspect its experts, then Escape to return. See statistics for paging and controls.

Options

The default is 4-bit experts with two adaptive speculative drafts.

Prepare low-bit expert stores ahead of inference:

target/release/cherenkov prepare flash --experts 3
target/release/cherenkov prepare flash --experts 2
target/release/cherenkov prepare flash --experts 2,3

Replace the indexed reference with a local checkpoint path to pack it directly. The 4-bit base is built if needed and retained; selected low-bit stores coexist beside it. Allow about 39 GB extra for 2-bit, 54 GB for 3-bit, or 93 GB for both. Existing stores are reused. Inference also builds a missing variant on first use.

Server configuration covers TOML defaults, memory limits, cache policy and reloads. See running options for the full interface.

How it works

These are active parts of the default engine:

• MTP speculation. The checkpoint’s own multi-token prediction head proposes up to two tokens. The trunk verifies them together and commits the accepted prefix, restoring recurrent state after a rejection. The first draft shares the trunk’s command buffer; a second is chained after full acceptance.
• Speculative routing. A one-block lookahead predicts which experts the next block will need and starts background reads. Actual routing still determines which experts run. Required misses take priority over speculative reads.
• Expert cache. An adaptive, resident LRU pool keeps recently used experts in unified memory. The GPU computes cached experts while CPU threads read missing records directly into free pool slots.
• Block address tables. Each block forms the union of experts needed by its token and draft rows. A table maps those experts to GPU cache addresses and tags their precision; separate per-row weights preserve each token’s routing. Kernels follow the table, so experts can change cache slots without moving the rest of the model.
• Shared page mappings. Packed dense weights are memory-mapped and exposed to Metal without a second copy. The expert pool also shares CPU/GPU pages: disk reads fill the same memory the kernels consume. Events keep the GPU from reading unfinished records and the CPU from overwriting active slots.
• Custom Metal kernels. Quantized projections, expert dispatch, sparse attention, DeltaNet, PLE and MTP run in native kernels. Longer prompts use batched matrix kernels and a bounded expert streaming ring.
• N-gram offloading. The large n-gram embedding table stays in an SSD-backed mapping. CPU threads prefetch the selected rows through the OS page cache, then dequantize them into small shared buffers for the GPU’s PLE blocks. Only those gathered embeddings occupy GPU buffers.
• Prefix caching in server mode. Repeated prompts restore attention, recurrent and MTP state, then process only the uncached suffix. Memory, entry count and idle expiry are bounded.

Lower-bit experts are opt-in. --experts 3 or --experts 2 compresses routed experts in both prefill and decode. With --miss-experts 2, Q4 lookahead reads continue normally, while unpredicted misses fetch smaller 2-bit records just in time. A fetched record keeps its precision while cached; the block table selects the matching kernel. Shared experts and dense projections retain their original precision. Lower-bit stores are derived once from Q4 and reused; no quantization happens in the decode loop.

See the engine guide for the address-table layout and synchronization. Direct file-backed expert residency is a separate developer option; the default uses the shared pool described above. The documentation index maps the remaining guides and source.

Development

Install the tools with mise install, then run:

mise run hooks       # install pre-commit checks
mise run check       # portable lints and Rust target/dead-code checks
mise run check:full  # also run tests, Metal validation, and the site build
mise run coverage    # instrumented tests and HTML/LCOV coverage reports
mise run fix         # apply Rust, spacing, and Markdown fixes

PRs run the same check groups on Ubuntu and macOS and save coverage reports. See validation for the groups, platform requirements, individual checks, and additional Clippy and Oxisym diagnostics.

License

MIT, with third-party notices. Model weights have their own license.

Documentation

Run mise run docs on macOS to build the website in _site/. .gitattributes selects the files and SUMMARY.md sets the navigation. Changes to main publish automatically.

• Rust API: generated library reference, including private items.
• Running: CLI, HTTP API, prefix caching and precision modes.
• Server configuration: TOML, memory policy and control CLI.
• Statistics: summaries, detailed JSON, and the live dashboard.
• Storage: paths, downloads and packed expert stores.
• Model index: named models, imports, retention and cleanup.
• Engine: model execution, expert streaming and synchronization.
• Developer options: diagnostic environment variables.
• Validation: tests, linters and known limitations.
• Serving architecture: ownership, scheduling and cancellation.
• Benchmarks: suite, reports and pelicans.
• Saved results: measurements and generated answers.

Repository layout

Model weights and packed stores use the configured data directory.

Building the documentation

Install Rust and mdBook with mise install rust github:rust-lang/mdBook, then run mise run docs (or cargo xtask docs). The build generates documentation for all workspace libraries, fails on rustdoc warnings, and publishes the complete reference under _site/api/ after building the book. It needs macOS for the Apple framework dependencies; model weights are not required.

To preview the combined site, run python3 -m http.server --directory _site 8000 and visit http://localhost:8000/. Rebuild to pick up source changes.

For just the Rust reference, run:

cargo doc --locked --workspace --no-deps --document-private-items --open

Rust API

Browse the generated reference for the workspace libraries:

• cherenkov: inference, model execution, storage, configuration, and serving.
• cherenkov-model-data: checkpoint formats, tensor metadata, byte access, and store discovery interfaces.
• xtask: repository automation, documentation builds, and benchmark tooling.

The reference includes private items to help contributors explore the internals. Private items are implementation details and can change without notice. Each reference has its own search and links to the Rust source. The book’s search covers the guide chapters.

The CLI is documented in Running. Cargo generates the library reference for each package; the binaries share their libraries’ names.

These links work on the published site or in a local site built with mise run docs. See building the documentation for details.

Running Cherenkov

See the README for installation and CLI examples.

HTTP server

cherenkov serve /path/to/model --port 8080 --max-ctx 4096

Connect clients to http://127.0.0.1:8080/v1 with model cherenkov. The server binds to localhost and does not authenticate HTTP requests. Clients that require an API key can use a placeholder.

Both completion endpoints support JSON and SSE, finish reasons, token usage, and stream_options.include_usage. Requests must use Content-Length; chunked request bodies are unsupported.

curl http://127.0.0.1:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"cherenkov","messages":[{"role":"user","content":"Explain hash collisions."}],"max_tokens":128,"stream":true}'

Generation options

These defaults can be set in TOML. Penalties count prompt and output tokens. Selection applies penalties, temperature, top-k, then top-p. Sampling or penalties disable MTP verification. CLI sampling uses the matching hyphenated flags, such as --top-k and --presence-penalty.

Both max_tokens and max_completion_tokens are accepted. Stop strings, JSON schemas, images, reasoning effort, template controls, and the Responses API are unsupported. Unsupported generation controls return an error. --raw, --check, and --repeat are CLI-only.

Tool calling

Chat also accepts tools and tool_choice. Tools must be function definitions (type function, a name matching [A-Za-z0-9_-]{1,64}, and an object parameters schema); strict: true and non-object schemas are rejected. tool_choice accepts auto (default) and none; required and a specific-function choice are rejected because the checkpoint cannot be obligated to call a function. parallel_tool_calls is accepted only when it is true while tools are enabled or absent.

The checkpoint renders the tools block and emits calls as XML-style markup. The server parses that markup incrementally and holds back bytes that could belong to a terminal marker. Text before the calls is preserved as content, with separator whitespace immediately before the marker removed. Parsed call markup is excluded from content in streaming responses, JSON responses, and retained session history. A call-only assistant message has content: null in JSON responses and session history; its stream contains no nonempty content delta. Streaming returns the calls together in an indexed tool_calls delta before the final finish-reason chunk.

Calls receive finish_reason: "tool_calls" when the model stops naturally. A token-budget cutoff keeps finish_reason: "length", even if calls were recovered. Recovery can retain a call whose name and parameters are complete but closing tags are missing, or keep earlier calls and discard a malformed or explanatory suffix. If the region cannot be recovered, its bytes are returned verbatim as content, including in streaming responses. Discarded suffixes and arguments that violate a declared parameter type produce a one-line diagnostic.

The client executes the calls and sends role: "tool" messages with the results. Retained sessions keep structured calls for the next turn without duplicating their markup in content. Session history stores decoded argument objects; API responses and fresh requests carry arguments as a JSON string.

Chat template

Chat accepts text messages with system, developer, user, assistant, and tool roles. Developer messages are mapped to system messages. A system message must come first, and the conversation must contain a user query.

The renderer loads chat_template.jinja, falling back to the string or named default template in tokenizer_config.json. It supplies messages, add_generation_prompt=true, and enable_thinking=false. The template controls formatting and message validation. Raw CLI prompts and text completions bypass it.

Preparation copies template metadata into the artifact. Re-running prepare fills missing files from an available local or retained source, preserving existing metadata. Repairs publish a new artifact; active readers keep their original files.

Sessions

curl http://127.0.0.1:8080/v1/sessions \
  -H 'Content-Type: application/json' \
  -d '{"temperature":0.7,"top_k":20,"seed":42,"context_tokens":2048}'

# Send only new messages when using a session ID.
curl http://127.0.0.1:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"session_id":"session-...","request_id":"turn-1","messages":[{"role":"user","content":"Explain hash tables."}],"max_tokens":128,"stream":true}'

curl -X POST http://127.0.0.1:8080/v1/requests/turn-1/cancel

Without session_id, send the full conversation. A session permits one in-flight turn. Different sessions progress in turn on the shared model.

Successful turns save messages, effective sampling settings, and RNG state. Omitted sampling fields inherit session settings. An explicit seed restarts the random stream; greedy turns consume no random draws. Sessions are held in RAM and are subject to retention limits. See statistics for session usage fields.

Cancellation discards the new turn, including its settings and RNG changes. Retry from the last committed history. Streamed text is provisional until completion. The current GPU step or prefill chunk finishes before cancellation takes effect. Cancellation cannot undo a turn once final publication commits it. Disconnects detected during writes and full output queues cancel unfinished work.

Active cancellation returns finish_reason: "cancelled". Cancellation before admission returns HTTP 499. Missing sessions return 404; busy sessions and duplicate request IDs return 409; exhausted request capacity returns 503.

A supplied request_id must contain 1–80 ASCII letters, digits, hyphens, or underscores and be unique among registered requests. Otherwise the server assigns one. It appears in completion IDs and the SSE X-Request-ID header.

context_tokens limits prompt, output, and speculative headroom within the server’s capacity. Lower limits reduce checkpoint reservations but do not resize the shared GPU buffers.

Prefix cache

The server restores the longest cached prompt prefix and processes the rest. Checkpoints contain attention, recurrent, convolution, PLE, token, and MTP state. They store model state, not responses, and disappear on server exit.

--prefix-cache-mb defaults to 512 MiB; 0 disables the cache. TOML also sets entry and idle limits. The cache evicts least recently used entries and skips oversized checkpoints. A short checkpoint uses about 113 MiB plus context storage, so the default budget holds roughly four short prefixes.

Checkpoints are saved at message boundaries, complete prompts, and selected prefill chunks. MTP’s following-token dependency is checked before reuse. Cached and fresh runs can choose different tokens on close argmax decisions because expert accumulation order can differ.

Reused tokens appear in usage.prompt_tokens_details.cached_tokens. For SSE, set stream_options.include_usage to receive usage in the final event.

Precision and memory

The default is 4-bit experts, two adaptive MTP drafts, and no deadline cut. --drafts 0 disables speculation and omits the draft head.

--experts 3 or --experts 2 applies to routed experts in prefill and decode. Mixed mode requires 4-bit residents and uses --miss-experts for fetched misses. Shared experts and dense weights keep their original precision. Lower precision changes output. A nonzero --cut-weak can skip late weak experts, making output depend on disk timing.

See storage for packing low-bit stores. Context uses about 22.5 KB per token, reducing the adaptive expert pool. --pool-gb N sets only that pool, in decimal GB; --pool-gb max uses more of the device’s budget. The runner checks prompt, requested output, and draft headroom against --max-ctx before loading the GPU model.

Timing

The CLI writes generated text to stdout and load, prefill, and decode timings to stderr. --max-tokens N --no-eos --repeat N measures repeated generation with loaded weights and fresh sequence state. --check runs the slow CPU reference and must not be used for speed measurements.

Warm repeats retain the expert pool and can change accumulation order. Use fresh processes with matching options for output comparisons. See the benchmark suite for measured workloads.

Server configuration

cherenkov serve /path/to/model --config cherenkov.example.toml
cherenkov status --json
cherenkov dash
cherenkov stats summary
cherenkov stats layers
cherenkov stats experts 0 --offset 0 --limit 64
cherenkov config show
cherenkov config reload

See statistics for summaries, detailed JSON, and dashboard controls.

The example TOML lists all settings. Precedence, from lowest to highest: built-in defaults, TOML, explicit CLI flags. Unknown keys and invalid combinations are errors.

Without --config, the server reads the default file described in storage. Set server.model_dir to a local path or server.model = "NAME" to an alias or source URI in the model index. The TOML keys are mutually exclusive. A positional CLI model path or serve --model NAME overrides either TOML selection. With no selection, the server selects the same pinned HF reference as prepare. Prepare it once before starting the server without a model argument.

Explicit relative paths in TOML resolve from the config file’s directory; CLI paths resolve from the working directory. Aliases and source URIs are preserved. Use absolute paths in TOML to avoid relying on shell expansion.

serve --print-config prints resolved settings without loading the model. Explicit CLI values override TOML even when equal to built-in defaults. Use --pool-gb adaptive or --no-eos=false to restore those defaults.

Reload

Only [defaults] and [defaults.sampling] are reloadable. Changes to [server], [limits], or [experts] require a restart and reject the whole reload. Reload rereads the startup file, reapplies startup CLI flags, and validates before replacing the configuration. It requires a startup config file. An unchanged reload keeps the same generation number.

Requests capture defaults before their bodies are read. Queued and active requests keep that snapshot. Request fields can override generation defaults within server limits; EOS policy stays server-owned. Sessions retain their committed sampling settings across reloads. --repack runs only at startup.

Memory and scheduling

memory_gb excludes general process overhead. Buffer allocations are checked at load and during prefill. Larger machines can use a larger explicit budget. Pool sizing reserves scratch for the configured chunk size and context. An explicit pool that leaves too little scratch capacity fails startup.

The worker runs one prefill chunk or complete decode step at a time. Switching requests copies sequence state to and from CPU memory. Weights, expert slots, and scratch remain shared.

prefill_quantum sets the maximum chunk size and its scratch reservation. Larger chunks reuse fetched experts across more tokens but leave less memory for the expert pool. The default of 128 preserves pool space on the 32 GB M4 MacBook Air used for benchmarks. On a larger Mac, try prefill_quantum = 4096 in [limits] and restart.

A lone request uses up to prefill_quantum tokens per chunk. When another request is active or queued, chunks start at min(128, prefill_quantum). The worker adjusts their size within that range to aim for prefill_chunk_seconds per chunk. Short tails, memory-limited chunks, and failed chunks do not affect this adjustment. Setting prefill_chunk_seconds = 0 disables pacing. Actual chunks may be smaller to fit available memory or prompt boundaries. cherenkov status --json reports the pacing target in tokens as prefill_chunk_tokens.

The duration is a target, not a deadline. Cancellation and newly arrived requests wait for the current chunk to finish.

To compare chunk sizes, run the same prompt at 128 and 4096, restarting between runs. Keep the pool setting unchanged if it fits both reservations. Collect cherenkov stats summary --json before and after each request and subtract the cumulative counters to get its chunk count, tokens, time, and read bytes.

Admission reserves checkpoint, sampling, token, and text storage for the requested context. Requests wait if they cannot fit together; one that cannot fit alone returns 503. A lower request context reduces this reservation but does not resize the GPU context buffers.

Prefix checkpoints and idle sessions use independent LRU eviction and expiry. Cache hits refresh expiry; successful turns refresh session retention. Queued, active, and publishing turns pin their sessions. Failed or cancelled turns preserve committed history and RNG state. Oversized prefix entries are skipped. Expiry runs before lookup and once per idle second; GPU work can delay release. Neither store spills to disk.

Each response writer has a 16-frame queue and a 30-second write timeout. Overflow or write failure cancels unfinished generation. Request IDs remain reserved through final output, bounding writer count. Queued requests can be cancelled before admission.

Expert policy

resident_bits selects 4, 3, or 2. miss_bits follows it unless specified; mixed mode requires 4-bit residents. The policy applies to routed experts in prefill and decode. Dense and shared-expert precision is unchanged. One low-bit layout is attached per engine, even when both variants exist on disk.

build_missing_store=true permits startup conversion. Set it to false to require a valid existing store. Precision never changes automatically on memory or IO failure. An infeasible explicit pool fails startup.

Adaptive sizing uses the device’s recommended working set minus host and fixed allocations. It leaves the larger of 6 GB or the scratch allowance outside the pool, and respects the configured memory budget.

cut_weak is off by default. A nonzero value permits skipping late weak experts and makes output depend on IO timing. Developer overrides apply independently of TOML.

Control socket

The default is TMPDIR/cherenkov-UID/control.sock. Server and CLI must use the same TMPDIR, or select an instance with --socket /absolute/path/control.sock. The server creates its private directory if needed; its parent must exist.

The directory must be owned by the server user with mode 0700; the socket uses 0600. Both peers check the effective UID with getpeereid. Processes running as that user are trusted. A held lock prevents duplicate owners and permits stale socket cleanup. Symlinks, unsafe permissions, and live sockets are rejected.

Control is separate from HTTP. Each connection carries one newline-terminated JSON request and response, limited to 64 KiB with a two-second input deadline. Operations are status, stats_summary, stats_layers, stats_experts, config_show, and config_reload. CLI errors return a nonzero exit status.

Statistics

These commands query the resident server through its local control socket. They do not load another model.

Add --socket PATH to select another server. While the model loads, status reports ready: false; stats returns a not-ready error and dash retries.

Summary and detailed output

stats prints a formatted summary by default. Layer and expert tables show selections, cache hits and misses, prefetches, and requested reads and bytes. Add --json for the complete response, including nested per-layer timing, prediction, and precision counters:

cherenkov stats summary
cherenkov stats layers --offset 0 --limit 8
cherenkov stats experts 7 --offset 0 --limit 32
cherenkov stats layers --offset 0 --limit 8 --json > layers.json
cherenkov stats summary --json > stats.json
cherenkov stats summary > stats.md
cherenkov status --json

Redirected text is Markdown. Terminal output uses the terminal palette; NO_COLOR disables styling. JSON retains integer byte counts and counters.

--offset selects the first entry; --limit sets the maximum returned (default 64, range 1–128). Pages can be shorter to fit the 64 KiB control frame. Follow next_offset until it is null. Each page includes its observation time; pages queried during generation may describe different observations.

Dashboard

dash refreshes once per second. The overview shows up to two minutes of expert read throughput, CPU read waits, and GPU stage coverage. Charts use changes between observations; row and detail panes show totals since load. Read-volume bars are scaled to the current page. Connection failures leave old values marked stale and gaps in the chart history.

Selecting a row updates its detail pane. Small terminals show only the focused pane; Tab reaches the others.

Counter meanings

Layer IDs index the packed expert table, including MTP entries. Counters include prefill, draft, and verification work, even when rows are later rolled back or cut.

Each layer’s streaming object contains prediction, phases, and quant. Quant entries identify their bits (4, 3, or 2) and separate prefill, demand, and prefetch reads. This distinguishes Q4 lookahead reads from low-bit fallback reads in mixed mode.

Reads record requested and completed bytes, successful and failed read counts, summed read duration, and maximum duration. Short reads count as failures; their bytes and duration remain included. Worker read durations can overlap.

Summary rates are averages since engine load. To measure an interval, divide changes in completed bytes or reads by the change in observation time.

Within each quant entry, cut_experts / eligible_weak_misses is the cut rate. cut_experts / selected_experts is the fraction of all selections cut. cut_batches counts affected batches. Cuts skip computation while the read finishes in the background.

Prefill chunks

The summary’s prefill object records completed batched-prefill work: chunks, tokens, min_chunk_tokens, max_chunk_tokens, and seconds. It also separates ring-reuse waits, n-gram gathering, and GPU command-buffer spans for DeltaNet, attention, experts, and MTP. GPU spans can include waits. Short prompts processed through the decode kernels are outside these counters.

Counters accumulate across requests and survive cancellation or rollback. seconds covers completed chunks; request prefill_seconds also includes checkpoint work and failed work. Expert-read bytes remain in the existing precision-specific prefill read counters.

Phase timing

Phase timing covers decode and MTP. The CPU and GPU intervals overlap.

CPU markers also record observation delay, preparation before resident release, and time after release. Read-worker CPU time is outside service_cpu_seconds.

gpu_stage_fraction is compute-pass time divided by the measured GPU timeline. gpu_handoff_gap_fraction is the remaining handoff time, including event and dispatch overhead. These measure phase coverage, not shader occupancy.

gpu_timing.status explains timer availability; failures include an error. gpu_timestamps_available indicates timer availability, and gpu_windows and invalid_gpu_windows count samples. Fractions are null without valid samples. The control thread reads copied statistics without waiting for the GPU.

Requests, sessions, and memory

status --json includes configuration generation, active request/session IDs, phases, token counts, timings, memory reservations, and cache/session usage. current identifies the last dispatched request.

Prefill time includes prompt processing and checkpoint creation. Decode time includes verification and enqueueing text. State switches and network writes are excluded. Generated-token totals include partial failed responses. Completed requests have finished writing; failed requests failed after queueing; rejected requests failed admission.

GET /v1/sessions/{id} includes committed turn count, history bytes, and cumulative usage: prompt, cached, and generated tokens plus prefill/decode seconds. History and usage commit together; cancelled turns add neither. Active-request usage is provisional until commit. Session eviction removes its statistics.

stats.memory contains the latest runtime observation:

KV/index capacity excludes recurrent state, PLE history, and scratch. These fields overlap and must not be summed. Prefix and session storage have separate stats.cache and stats.sessions counters. Observations include uptime and are published after load, during generation, and while idle.

CLI generation emits the same fields in a memory_stats JSON line on stderr. Benchmark reports store that object in metrics.memory; older reports may leave it null.

Storage and downloads

cherenkov paths prints storage paths and the default model reference. downloaded_model is the path used by the deprecated download command.

These defaults also apply on macOS. Each XDG override must be absolute; Cherenkov appends cherenkov to it. Empty or relative values are ignored. Resolving paths does not create directories. Downloading and packing create the required parent directories.

--root DIR places data in DIR, scratch in DIR/scratch, and configuration in DIR/cherenkov.toml. Put it after the subcommand:

cherenkov paths --root /Volumes/Models/cherenkov

An explicit model path overrides the managed default. serve reads the default config if present, or the file selected by --config. Its [server].root can change model lookup; a CLI root takes precedence. Other subcommands do not read TOML. See server configuration for path resolution.

Use the model index to name checkpoints, prepare HF sources without retaining their downloads, and collect unused managed stores.

Prepared storage

data/
  index.json
  artifacts/<id>/
    config.json
    tokenizer.json
    manifest.json
    dense.bin
    ngram.bin
    experts.bin
    experts2.bin + manifest2.json
    experts3.bin + manifest3.json
  leases/<id>

Prepared stores and retained source downloads live under artifacts/. External checkpoints stay in their original locations. Clearing transfer scratch leaves prepared stores intact. model gc removes unreferenced owned artifacts; see the index layout.

Deprecated download command

The getting-started workflow uses prepare SOURCE to create an indexed store. The standalone download command is deprecated but still works. Use prepare --keep-source to retain downloaded weights.

The default checkpoint is Sawfwair/Qwen3.8-Flash-Next-MLX-4bit at 6cc9bbc0fae9ce26b7670b3ed1e26d557c154506. Branches and tags passed to download --revision are resolved to full commits. Other downloads print their model path; pass it to prepare to register and prepare it.

download --metadata-only fetches config and tokenizer files without weights. A later full download reuses them. The downloader validates the architecture, reads shard names from the safetensors index, and checks available disk space. This checkpoint needs about 104 GB for source weights and another 104 GB for base packing.

Downloads use the Rust hf-hub client and Xet. Supply credentials through HF_TOKEN, an existing Hugging Face login, or download --hf-token. HF_TOKEN avoids exposing a token in command arguments. Tokens are not saved in Cherenkov config or manifests.

Xet scratch uses scratch/xet/, unless HF_XET_CACHE is set. Credential lookup follows HF_TOKEN, HF_TOKEN_PATH, and HF_HOME. Authentication grants account access and rate limits; it does not guarantee faster transfers.

Prepare

cherenkov prepare /path/to/model
cherenkov prepare /path/to/model --experts 2,3

Preparation publishes an indexed artifact under data/artifacts/. --output DIR exports to a new directory and records it in the index. An existing packed directory can also be registered as input. Omitting the source selects the default HF checkpoint, also selected by bare serve. pack remains an alias for prepare.

--experts accepts 4, 3, or 2, separated by commas, spaces, or repeated flags. The default is 4. Low-bit targets require the Q4 base, built first if absent. Missing variants share one pass through the base records. Valid stores are reused; unselected stores are left intact. Allow about 39 GB for 2-bit and 54 GB for 3-bit in addition to the base store.

The packer checks disk space and publishes manifests after flushing output. Inference also builds missing low-bit stores. --repack during inference rebuilds the selected low-bit store.

The server’s prefix cache and sessions are held in RAM. They have no disk store.

BF16 import

prepare also accepts native Hugging Face BF16 checkpoints for qwen4_exp. It splits fused expert matrices and quantizes them directly into the Q4 store, without an intermediate checkpoint. Routers, convolutions, and norm vectors remain BF16. The packer folds zero-centered norm offsets into their weights. N-gram quantization groups follow the row width.

cherenkov prepare /path/to/bf16-model --experts 4,3,2

The packed directory keeps the tokenizer and configuration. If the checkpoint has no MTP weights, its packed configuration disables drafting; the source configuration stays unchanged. Existing MTP weights are retained.

Checkpoint inspection

cherenkov inspect /path/to/model
cherenkov inspect /path/to/model/packed
cherenkov inspect /path/to/model.gguf --json

inspect --json reports tensor shapes, encodings, byte ranges, and preparation requirements for the Qwen4-exp engine. Inspection uses the prepared artifact when one is registered; otherwise it uses the source description.

The cherenkov-model-data workspace crate reads safetensors and GGUF without Metal. Safetensors is a container; MLX affine quantization is a separate encoding convention. The MLX adapter combines codes, scales, and biases into logical tensor descriptions. GGUF block formats retain their own encodings. The Qwen adapter assigns tensor roles and describes n-gram hashing and shards.

Sources expose object IDs, sizes, streaming byte reads, and optional mappings. IDs are local to a source; callers must rebind them when copying to another store. Mapped views keep their mappings alive after the source is dropped. Callers must keep the files unchanged and, for indexed stores, hold the artifact lease until all views are finished. The lease protects the store from garbage collection. Strides describe expert records and interleaved n-gram rows without copying them.

The reader preserves configuration, shard metadata, and unknown encodings. Inspection does not imply execution support: other architectures and GGUF conversion remain unsupported by the Qwen4-exp engine. This layer does not allocate GPU pools or manage model ownership and garbage collection.

Model index

The index records model sources and prepared stores. A source can be a local checkpoint, an HF cache snapshot, a packed directory, or a Hugging Face repository. Registration inspects the model; it does not imply execution support.

Select a model

cherenkov prepare hf://owner/repo@revision --name small-moe
cherenkov serve --model small-moe
cherenkov model list
cherenkov inspect small-moe --json

prepare resolves the source through the index, downloads weights if needed, and prepares it. model add SOURCE registers metadata without preparing weights. An alias is optional. All model commands use the same selectors:

The owner remains part of the identity. An unqualified HF URI reuses its indexed commit without contacting HF. If several commits match, specify @commit, an alias, or an ID. Commit prefixes need at least eight hexadecimal characters and must select one entry. Cached commit prefixes follow the same rule. Branches and tags are resolved when explicitly requested; use the returned reference or alias for later index operations. Bare names only look up aliases and IDs; they never trigger a download.

URLs use literal repository names; queries, fragments, escaped names, and dot segments are rejected. A revision may contain slashes, such as @feature/branch. Quote filesystem paths containing spaces. Legacy model: selectors and the pack command remain accepted for compatibility.

Filesystem stores

cherenkov store add models /Volumes/Models
cherenkov store add cache ~/.cache/huggingface/hub --layout hf-cache
cherenkov prepare disk://models/owner/repo --name local-model
cherenkov prepare disk://cache/owner/repo@commit
cherenkov store list --json
cherenkov store disable models
cherenkov store enable models
cherenkov store remove models

The default layout is ROOT/owner/repo. hf-cache reads HF’s models--owner--repo/snapshots/commit layout and cached refs without downloading. Registration records the root without crawling it. Disabling a store blocks its disk:// selectors; existing aliases and prepared models remain usable. Removing a store deletes only its registration. Re-registering its name creates a new store ID. Local files always remain externally owned.

Register and inspect

HF registration pins the requested revision (default main) to a full commit. It reads config, the shard index, and each safetensors header using byte ranges. Small n-gram metadata arrays may also be read, but full shards remain remote. Servers that ignore range requests are rejected. Remote GGUF registration is not supported.

Inspection reads up to eight shard headers concurrently. Color terminals show a spinner during metadata resolution and a header progress bar with an estimated remaining time, using the same Ratatui theme as reports and the dashboard. Redirected stderr, NO_COLOR, and terminals without color use plain text updates. Progress stays on stderr, including with --json; stdout holds the result. Shard names come from the weight map or repository listing.

Credentials come from --hf-token, HF_TOKEN, or the HF token file. The index stores the repository, endpoint, and commit, but no credentials.

model list and inspection of existing entries work offline. Both accept --json, including a resolvable reference field. Text output uses the terminal’s report renderer. inspect --json includes tensor shapes, encodings, and byte ranges. Inspection supports more architectures than the inference engine. For an indexed model, inspect describes its prepared store if one is recorded; otherwise it returns the source description saved at registration.

Prepare and run

cherenkov prepare small-moe --experts 4,3,2
cherenkov small-moe "Explain this model."
cherenkov serve --model small-moe

Indexed packing writes a managed store. When an HF source is needed, packing reuses its retained copy or downloads it to a temporary owned directory. New downloads are removed after successful packing unless --keep-source is set. Conversion needs room for the complete source and prepared output together; it does not release shards as it proceeds. prepare --hf-token supplies credentials when the registered source requires them.

Local sources and existing HF cache snapshots remain externally owned. Cherenkov never deletes them. prepare --output DIR exports to a new, externally retained directory outside the managed artifacts/ directory and records it in the index. Direct paths also resolve through the index. The deprecated download command keeps its older layout; prepare can register and use its output.

Generation and serving require a prepared model. Missing expert variants are built according to the existing packing policy. Preparation also fills missing auxiliary metadata from an available local or retained source. Older model/packed layouts can keep config and tokenizer in the model directory. Each update publishes a new store, so running readers keep using their original files. Managed stores reuse unchanged binary files through hard links where possible. Variant availability checks load the manifest and check for the Q4 base file. Low-bit checks also validate layouts, sizes, and sample records. Automatic repairs follow the same build policy as missing variants and preserve files used by existing readers.

A server can select an indexed model in TOML:

[server]
model = "small-moe"

server.model and server.model_dir are mutually exclusive. CLI model selection replaces the TOML selection. Changing either setting requires a server restart.

Remove and collect

cherenkov model remove small-moe --source-only
cherenkov model gc --dry-run
cherenkov model gc
cherenkov model remove small-moe

--source-only releases a retained source copy after checking the prepared weights and configuration. Plain remove removes the index entry. Neither command deletes files. gc collects unreferenced owned directories and abandoned imports. External directories are left in place. Live leases protect stores in use by readers or imports. These commands also accept --json.

Reported bytes are file lengths, not allocated disk blocks. Hard-linked files can appear in more than one store’s total. External-byte totals cover registered artifact directories, excluding raw local sources.

Storage and interfaces

data/
  index.json           model identities, sources and artifact references
  index.lock           catalog lock
  artifacts/<id>/      owned prepared stores or retained source downloads
  leases/<id>          reader/import locks

The catalog is replaced atomically. Ownership is recorded explicitly; a local folder does not become owned because it lies under the data root. Lock files remain after collection so concurrent processes use the same lock identity. Registering an existing managed entry reuses its ownership record. Other paths inside managed artifacts, including private HF snapshot paths, are rejected. See storage for root selection.

ModelIndex manages registration, resolution, packing, and collection. ArtifactLease keeps a resolved directory live while its byte sources, mappings, and GPU views are used. The cherenkov-model-data crate describes containers, tensor encodings, byte reads, and optional mappings without Metal dependencies. HF header discovery implements that same byte-source interface.

Library callers can set ResolveOptions.events to receive owned ModelEvent values during remote inspection. Callbacks run serially on the calling thread, outside the catalog lock. Keep callbacks brief, or forward events to a channel. Cached lookups emit no inspection events; the returned Result reports success or failure.

StoreDiscovery adapters advertise search and enumeration separately, with their own typed filters such as HF’s author. The Discovery dispatcher validates requests before calling an adapter and attaches a stable store ID to each result. Continuation cursors belong to that store and request, including its page size. Results carry optional common metadata and provider-specific fields. Adapters exclude unknown values from filter matches and report known metadata gaps. Search adapters and CLI search commands are not yet implemented.

The current backend manages complete directories. Cross-model object deduplication, independent n-gram artifacts, and shard-at-a-time payload conversion remain future work.

Engine

The Metal engine streams routed experts through a bounded cache. Dense weights remain resident, and the CPU gathers n-gram rows from a file mapping. The CPU path is the reference used by --check. Neither path requires MLX at runtime.

Source map

Decode and prefill have child modules for attention, DeltaNet, experts, hyper-connections, PLE, and sampling. Tests mirror the GPU modules under tests/unit/qwen4_exp/gpu/. See the kernel map for shaders.

Host repr(C) structs must match their Metal definitions in field order and type. Buffer-binding offsets are bytes; kernel tensor indices are elements unless stated otherwise.

Decode and expert IO

A step processes one token and up to three drafts in row-major buffers. The residual has four hidden-width streams. MoE output may be deferred until the next normalization; PLE applies any pending output before its transform.

Each block uses this handshake. Values are relative to its sequence number.

Address tables

The CPU forms the union of experts selected by all token rows and resolves each (layer, expert) pair to a cache slot. slot_tab holds GPU addresses; wmap holds each row’s routing weights, with zero for unused experts. Entries are ordered by residency, then by maximum routing weight across rows. The final two table entries hold resident and total expert counts.

Address bit 63 marks Q3 and bit 62 marks Q2. Kernels mask these tags before loading a record and use them to select dequantization.

For a token routed to A/B and a draft routed to B/C, with B missing:

The GPU computes resident contributions while the CPU fills missing slots. Current-step slots cannot be evicted. With cut_weak, late weak experts may be removed from the active table, but their slots remain reserved until IO completes. Output then depends on read timing.

Lookahead predicts the next block’s routes. Its reads start after the current block’s required misses finish. Residency changes accumulation order and can change rounding, so use fresh processes for output comparisons.

Prefill

Long prompts run layer by layer with matrix projections. Tokens are grouped by expert, gathered, projected, and scattered back. Frequently used experts stay in the decode pool; others stream through a 64-record ring in groups of eight.

The GPU signals after consuming a group. The CPU waits before reusing its slots, fills the next group, then signals readiness. Each prefill event has one writer and increasing values.

Prefill and decode share record layouts and stores. Prefill uses Q4 or Q2/Q3 GEMMs with 8/16/32-token tiles, half weights, and float accumulation. Decode uses specialized half-dot reductions.

Uniform low-bit mode fills both pool and ring from that store. Mixed mode keeps resident precision, adds kept records at Q4, and reads transient misses at --miss-experts precision. Ring slots retain Q4-sized spacing. Shared experts and dense projections remain Q4. Deadline cuts apply only to decode. The ring-wait metric measures CPU waiting for GPU consumption, not disk IO.

State and rollback

The CPU stores each DeltaNet head as S[key_lane][value_lane]. After normalizing q/k and applying the causal convolution, each token computes:

1. S = decay * S
2. delta = beta * (v - S^T k)
3. S = S + k delta^T
4. y = S^T q, then gated RMS normalization

The GPU stores the transpose. One simdgroup owns a value lane and distributes 128 key lanes across 32 threads, keeping four floats per thread in registers. Reduction order and half projections can differ from the CPU reference.

Verification saves recurrent state and convolution history after candidate rows. After partial acceptance, commit(n) restores row n - 1 and advances the logical position. Attention and PLE entries beyond it are ignored or overwritten. MTP tracks its own KV length and following-token dependency; prefix reuse checks both.

Preserve reduction order, accumulator precision, and unrolling when moving code. These affect numerics and register pressure. MTP’s expanded residual uses an eight-row projection path even though verification has at most four rows.

Memory ownership

The packed model owns dense mappings borrowed by Gpu. Expert pools own their slot memory. Shared CPU/GPU pages require event or command-buffer synchronization before reads or overwrites.

The default expert pool is a shared Metal allocation. CPU file reads fill the same slots the GPU consumes. Dense weights use newBufferWithBytesNoCopy over a file mapping. CHERENKOV_POOL=set instead wraps mapped expert regions and supports Q4 only.

The n-gram store remains CPU-mapped. Token history determines rows to prefetch at each decode step or prefill chunk. Packed::ngram_row dequantizes them into small shared buffers; the GPU runs PLE projections, gating, and convolution. The full table need not remain GPU-resident.

Metal kernels

common/ contains shared primitives; qwen4_exp/ contains model kernels. device/clock.metal is the dependent-FMA throttling probe. Rust files contain no embedded Metal kernels.

Compilation and dependencies

src/kernels.rs assembles two libraries and the probe for both the engine and tests. Fragments cannot compile independently. The assembler supplies metal_stdlib, the namespace, and #line directives.

Shared types precede their users; Q4 row helpers precede hyper-connections and experts. Q2/Q3 and address-table helpers are private to experts.metal. Helpers do not cross library boundaries.

Keep macros with their definitions and instantiations. Preserve the matching lane and reduction order of the Q4 row helpers. Changes to projection precision or batching require numerical and performance checks.

Editor diagnostics

mise run clangd generates kernels/.clangd. It supplies the editor prelude and force-includes preceding fragments. check-metal rejects stale config. clangd parses MSL as C++; use the check-metal VS Code task for compiler errors.

Do not pass --compile-commands-dir to the Metal language server. It overrides the generated config and can break relative includes.

Developer environment reference

Unset these overrides for normal inference. Fake reads and skipped stages produce invalid outputs.

CHERENKOV_MODEL_DIR selects the packed model for real-weight tests. Otherwise they use the managed model, or skip if it is absent. Other CPU and kernel tests need no checkpoint.

Validation

Install the pinned tools once, then choose a check group. Run native tests, coverage, and Metal checks without concurrent inference or benchmarks:

mise install
mise run hooks
mise run check
mise run check:full

CI and pre-commit call the same tasks in mise.toml. The full group runs unit tests once, then Metal checks and the site build; lightweight portable checks can run in parallel. Coverage remains separate because it runs an instrumented test suite. Run individual checks with check:fmt, check:spacing, check:markdown, check:workflows, check:hooks, check:tasks, check:rust, or check:metal.

Checks report failures without changing source files or installing tools. Rerun mise install when tool pins change. Rust compilation commands use --locked consistently; set CARGO_NET_OFFLINE=true for offline checks after dependencies have been downloaded.

The additional check:clippy and check:oxisym diagnostics are separate from the passing groups because they still report existing findings described below. Oxisym selects its own nightly toolchain and requires two additional tools:

cargo install --locked cargo-dylint dylint-link

Review automatic spacing changes: the rules group syntax, not meaning. Existing task names such as fmt-check, lint-md, check-metal, clippy, oxisym, fix-spacing, and fix-md remain aliases. The compatibility command lint-spacing runs check:fmt before check:spacing.

Tests

Engine unit tests are child modules in tests/unit/, mirroring the source. Container, byte-source, and discovery tests are in crates/model-data/tests/. Automation tests are in xtask/tests/.

Pull requests run portable pre-commit checks on Ubuntu and check all workspace targets and run the test suite on macOS 15 and 26. The macOS jobs include synthetic Metal tests, then explicitly check Metal syntax and the generated clangd configuration. Real-weight checks need a local model.

GitHub Actions caches Cargo downloads and compiled dependencies after successful runs. Test caches are separate for macOS 15 and 26; docs, coverage, and author checks use separate caches. Rust versions, Cargo manifests, and lockfiles contribute to cache keys. Workspace crates are rebuilt, and the docs build regenerates the API reference while reusing dependencies from target/site-rustdoc. The first run for a new cache is cold; later compatible runs can restore it.

Read the contribution terms in AUTHORS, then acknowledge them by adding your own entry as Name <git-email> (@github-login). You can do this in your first PR and use a GitHub noreply address. Later PRs reuse that entry. The check matches the PR author’s GitHub login; reviewers confirm that new entries were added by the contributors themselves.

Run the same check locally with cargo xtask check-author YOUR_GITHUB_LOGIN.

Metal tests require Apple Silicon. Real-weight tests use CHERENKOV_MODEL_DIR or the managed model and skip when it is absent. Low-bit checks skip absent Q2/Q3 stores. check-metal compiles the engine’s assembled libraries and rejects stale kernels/.clangd.

Prompt tests compare 20 independent template references. Text and error checks need no model. Token checks need the matching tokenizer, but no weights or GPU. See fixture provenance.

Coverage

Run mise run coverage on Apple Silicon to run instrumented workspace tests serially and generate reports in target/coverage/:

• html/index.html: browse coverage by file and source line.
• lcov.info: import coverage into an editor or another reporting tool.
• summary.txt: per-file and overall coverage totals.

Use mise run coverage:report to regenerate reports without rerunning tests.

Install the tools with mise install rust cargo:cargo-llvm-cov. If Rust is already installed without coverage support, run rustup component add llvm-tools-preview.

The Rust coverage job runs on macOS 26 for pull requests and main. Each run records its summary in GitHub Actions and retains the reports for 90 days in a rust-coverage-<commit> artifact. Compare run summaries to track coverage changes. There is no minimum coverage threshold yet.

Coverage measures Rust lines, regions, and functions. Test source files are excluded from the totals. It does not measure Metal shader execution or doctests. CI has no model weights, so model-dependent tests skip; their unexecuted Rust paths still count toward coverage. Local runs with model weights may cover more.

Pre-commit hooks

After mise install, run mise run hooks once to install the Git pre-commit hook. It checks Rust formatting, statement spacing, Markdown, workflow syntax, and all Rust targets with dead code denied. Checks run when matching files are staged; they report failures without modifying files. Rust checks require macOS.

Run mise run pre-commit to check all tracked files before opening a PR. The Ubuntu CI job runs the portable group; the macOS jobs run the native group followed by check:metal. Coverage and the test suite run separately from the commit hook.

Linked Git worktrees share the installed hook. Installation allows a missing configuration so worktrees on branches without .pre-commit-config.yaml can still commit normally.

Live checks

cargo xtask smoke server --model /path/to/model
cargo xtask smoke control --model /path/to/model
cargo xtask smoke sessions --model /path/to/model
cargo xtask smoke download

Each server check starts and stops its own process. Checks cover JSON/SSE, prefix reuse, reloads, concurrent sessions, cancellation, rollback, limits, and recovery. The download check fetches metadata only and checks reuse. Reports go to ignored results/*-smoke.json files.

Lint limits

All workspace packages deny Rust’s dead_code lint. mise run check:rust runs cargo check --locked --workspace --all-targets, including libraries, binaries, and tests. Unused private items fail the check. Public library APIs may be used by downstream crates, so this does not detect every unused public API.

Clippy uses cognitive complexity 12 and nesting 2, and checks unnecessary else branches. Use guard clauses and helpers without obscuring numerical or dispatch order.

Complexity passes. The experimental nesting limit still flags existing code. To run the other lints independently:

cargo clippy --workspace --all-targets --offline -- -D warnings -A clippy::excessive_nesting

Oxisym reports existing structural-similarity findings for manual review.

Known gaps

Use the benchmark suite for performance checks. Saved results apply to their recorded revisions.

• A local AC comparison found a possible 6.5% long-prompt decode regression after readability changes. It remains unresolved.
• Full download throughput and a fresh full-size pack have not been validated.
• Cached and fresh runs can differ on close argmax decisions as expert accumulation order changes. Exact state restoration does not prevent this.

BF16 fixture

The optional Git LFS fixture exercises BF16 packing, Metal prefill and decode against the CPU reference, and CLI generation. It also checks mapped views, indexed source removal, variant replacement while readers hold leases, and external exports after collection. The suite also covers store add, prepare, and generation by alias. It includes DeltaNet, sparse attention, routed experts, and n-grams; it has no MTP weights. Full-size BF16 import has not been run locally.

Loading checks

The model-data tests run without Metal:

cargo test -p cherenkov-model-data
cargo test -p cherenkov-model-data --test remote -- --ignored --test-threads=3

The remote tests read every safetensors header at pinned Hugging Face revisions and check it against the shard index. They fetch byte ranges only, with no weight downloads. All six cases passed locally:

These checks validate metadata and byte ranges, not weight contents or inference. DeepSeek’s quantization config and stored components are retained; its FP4/FP8 decoding is not implemented. The local MoE fixture additionally tests mapped weights, native-to-packed conversion, and CPU/Metal agreement. The test definitions contain the repository names and full revisions.

Prompt fixtures

chat_template.jinja is copied unchanged from the checkpoint. references.json contains 20 contexts, expected text or errors, token IDs, input hashes, and generator versions.

References were generated with Transformers 4.57.6, Jinja2 3.1.6, and tokenizers 0.22.2. To regenerate:

1. Use the recorded checkpoint and versions.
2. Render each saved context with Transformers’ _compile_jinja_template.
3. Check nonempty successful cases with AutoTokenizer.apply_chat_template, using both tokenize=False and tokenize=True.
4. Replace the expected text, token IDs, or error. Use add_special_tokens=False for tokenization. Do not use Cherenkov to generate reference outputs.

Empty-message cases call the template directly because the Transformers wrapper indexes the first message before rendering.

Cargo tests need no Python. Text and error checks always run. Token checks require CHERENKOV_MODEL_DIR with the matching tokenizer; weights and GPU are not needed. Hash checks reject stale inputs.

Qwen4-exp test checkpoint

This BF16 fixture is inference-optimization/Qwen3.8-Flash-Next-0.2B-A0.2B at revision 5cdc1eff790ad299680eda7b97241068224581e4 (MIT). It has four decoder layers and no MTP weights. The weights and tokenizer are stored in Git LFS; the other files are ordinary Git files.

mise install
mise exec -- git lfs install --local
mise exec -- git lfs pull
cargo test --release --test import -- --ignored --test-threads=1

The tests pack into temporary directories, compare Metal prefill and decode with the CPU reference, and run CLI generation. They also check mapped tensor views and indexed-store ownership through source removal, variant replacement, and garbage collection. The full fixture suite requires Apple Silicon. The default unit suite does not load it.

To clone without fetching the fixture, set GIT_LFS_SKIP_SMUDGE=1 for the clone.

Serving architecture

See running for the API and configuration for limits and defaults.

Source map

src/server.rs starts the server. Its child modules are:

src/prompt.rs compiles the checkpoint template once. Cache boundaries must match prefixes of the complete rendered prompt. Sessions pass typed settings and messages to the worker.

Ownership and scheduling

The worker runs requests round-robin, yielding after one prefill chunk or complete decode step. Switching requests saves sequence state to CPU vectors and restores the next request. A single active request stays on the GPU. Events and expert residency are shared and are never restored from checkpoints. Requests are interleaved, not batched together on the GPU.

Checkpoints contain KV and QSA state, DeltaNet state, convolution/PLE history, tokens, position, and MTP state. Scratch is reusable after GPU completion. Admission reserves checkpoint growth and sampler workspace; requests wait when their combined reservations cannot fit.

Sampling

Each request owns its RNG. Selection applies occurrence penalties, temperature, top-k, then top-p. Greedy, unpenalized generation uses GPU argmax and adaptive MTP. Sampling or penalties disable MTP verification.

A suspended request keeps its selected next token. It is not sampled again, and other requests cannot consume its random draws. Output-length boundaries do not draw unused tokens. Sessions continue their committed RNG unless given a new seed; greedy turns consume no draws. Penalty counts come from the full rendered prompt.

Exact-prefix hits return logits for the request’s own sampler. Cache reuse checks draft compatibility and MTP’s following-token dependency. It never reuses another request’s sampled token or RNG.

Cancellation and commit

Registration reserves an ID through final output. Cancellation can be set without entering the GPU worker. The worker checks before admission and work, and after prefill chunks. In-flight GPU work finishes before state is reused.

A turn pins committed history, settings, and RNG while generating provisional output. Preparing completion reserves space for the new history. Immediately before the final response, the writer atomically chooses completion or cancellation. Only completion commits the prepared session state.

Cancellation or failure releases the reservation and leaves the previous turn intact. A retry starts there. Disconnects after commit cannot undo the turn. Write failures, full queues, and socket timeouts cancel unfinished requests.

Retention

Sessions and prefix checkpoints have separate byte, count, and idle limits. In-flight sessions are pinned; idle sessions may be evicted. History can survive checkpoint eviction and be prefilled again. Neither store persists across server restarts.

GPU context buffers are allocated once at the server maximum. A smaller request limit reduces admission and checkpoint reservations, not GPU buffers.

Limits and tests

Tool calling uses the checkpoint template and XML-style output parser. The client executes the returned calls; sessions retain them for later turns. The worker tracks raw decoded bytes for reconciliation and response limits, and accumulates parsed content separately. Final tokenizer bytes pass through the parser before it closes. Streaming text, JSON content, and committed session content use the same accumulated text. Recovered calls preserve a length finish reason when generation reaches its token budget. See tool calling for supported request controls.

Effort controls, the Responses API, sampled MTP, and cross-request GPU batching are unsupported. Chat uses the checkpoint template with thinking disabled.

Tests cover sampler continuation, pending tokens, commit/cancel ordering, retention, memory admission, state restoration, and interleaved HTTP streams. Worker tests also cover tool-call content across streaming, JSON, and sessions, fallback delivery, final tokenizer bytes, and truncation finish reasons. See validation for commands. Concurrency throughput has not been established by these correctness tests.

Benchmarks

The runner accepts an indexed alias, source URI, or prepared directory. It uses the index to find the artifact and accepts both flat stores and older model/packed layouts. Use --root DIR if the model is in a separate index. Model inspection runs outside the timed inference capture.

cargo xtask bench flash --build-stores --update-readme
cargo xtask bench disk://models/owner/repo --root /path/to/index

The suite requires Rust, Metal, and a packed checkpoint. Run on AC power with other inference, builds, and GPU tests stopped. Prefix commands with mise exec -- if Mise is not active in your shell.

cargo xtask bench /path/to/model --dry-run
cargo xtask bench /path/to/model --build-stores
cargo xtask bench /path/to/model --cases code,prose --rounds 1
cargo xtask bench /path/to/model --output results/comparison
cargo xtask bench /path/to/model --output results/comparison --resume
cargo xtask bench /path/to/model --mode light --note "M5 Pro 48 GB, idle"
cargo xtask bench /path/to/model --mode heavy --build-stores --note "M5 Pro 48 GB, idle"

The runner builds offline unless given --binary. --build-stores permits missing low-bit stores to be built during load. Otherwise missing stores are an error. Allow an extra 39 GB for Q2 and 54 GB for Q3. Loading and conversion are excluded from prefill and decode timings.

Sharing a run

Two presets exist for sending a run to someone else. --mode light runs one round of the code, prose, and long-prefill cases on whichever expert stores are already built, about ten samples, and takes a few minutes. --mode heavy runs the whole suite including pelicans, which takes hours and needs every store or --build-stores. Both zip the finished results directory beside itself (--archive does the same for any run), so one file holds report.json, summary.md, the gallery, every answer, and the pelicans. Attach it to a pull request or issue. Explicit --configs, --cases, and --rounds override a preset’s choices.

Light mode consults the model index, including for --dry-run.

Each report records the machine in provenance.hardware_detail: kernel, memory, CPU thread count, and the capacity and free space of the volume holding the model, all read through libc; on macOS also the chip, GPU core count, OS version, and NVMe model and capacity from system_profiler; and a two-gigabyte uncached read sample of the expert store in GB/s taken before the first sample. Automatic machine metadata excludes serial numbers, device identifiers, the hostname, and home-directory paths. The runner’s binary and model arguments appear as <binary> and <model>; the runner’s --root path appears as <root>. Custom suite prompts and configuration arguments, generated answers, and --note text are preserved verbatim. --note records conditions such as power or other load. settings records the context capacity and cap overrides; the suite path appears as <suite> and its content hash is recorded in the signature. The server TOML is never read, and the pool is adaptive unless a configuration passes --pool-gb. The summary’s first line repeats the hardware facts so runs from different machines can sit side by side.

Suite

suite.json defines 80 fresh-process samples:

• Six completion workloads × four settings × three rounds.
• One long-prefill workload and one pelican per setting.

Settings are Q4, Q4 residents with Q2 misses and cut 0.08, Q3, and Q2. All use two adaptive drafts, an adaptive expert pool, and 8,192-token context. The mixed setting’s cut makes output timing-dependent. Configurations rotate between rounds; pelicans run last.

Answers run to EOS. Safety caps are 4,096 tokens, except LRU, reasoning, and pelicans at 7,168, and document summaries at 1,024. Capped or cycling answers are saved but excluded from medians. Completion does not establish correctness. Compare answer length and completion time alongside tg/s.

Expert precision applies to prefill and decode. The saved September 9 baseline used Q4 batched prefill in every mode; later prefill comparisons are separate.

Measurement

Each sample uses a fresh process without --check, warm repeats, prefix caching, or developer overrides. Power is checked at process boundaries and every 30 seconds. A power-source change invalidates the sample; shorter transitions may be missed.

The engine sizes its expert pool from Metal’s recommended working set after fixed buffers and reservations, and fits prefill chunks to available memory. The suite keeps prompts, answer caps, and context capacities fixed for comparison across machines. Custom suites can change those workloads; --case-cap changes answer caps and --mode light reduces the sample count.

Reported GPU memory above the target machine’s physical memory invalidates a sample. Memory is read after prefill scratch is released. gpu_span_ms is the interval between the command buffer’s GPU start and end timestamps, including gaps. io_wait_ms measures host servicing of expert reads. The intervals overlap and must not be added together. Neither measures GPU utilization.

Reports and resume

Browse the saved reports on GitHub or the documentation site. Published runs retain the report, summary, observations, outputs, and pelicans. The HTML gallery is a local preview.

Results default to results/<UTC timestamp>/ and are written after each sample. New runs are ignored by Git. Use git add -f results/<name> to retain one. SVG rates are reported separately; malformed or incomplete SVGs are invalid.

--resume requires matching binaries, indexed model identity, metadata, suite, and selections. Reports store the model ID rather than its local path. Older reports can migrate when their path or path hash matches the indexed local source. Reports containing only <model> without an identity require a new run. It skips successful and content-invalid samples. Engine errors, memory-limit failures, and power changes stop the suite and are retried on resume.

To retry capped answers, raise their caps with, for example, --case-cap code-lru=7168 --resume. Completed EOS samples remain. Earlier attempts and signatures are retained. Lower caps or other setup changes are rejected.

Every 30 seconds, cycle detection looks for four exact repetitions of a 32–512-word block at the output tail. A match saves the evidence, ends that sample, and continues the suite. The check does not detect every kind of loop.

Regenerate reports without inference:

cargo xtask summarize results/<name>
cargo xtask readme results/<name>

summarize preserves the run’s README. --update-readme runs the second command after a completed benchmark.

Paired prefill

cargo xtask prefill --before /path/to/old/cherenkov \
  --after target/release/cherenkov --model /path/to/model \
  --out results/prefill-comparison --rounds 2

This compares three prompt lengths with Q4/Q3/Q2 experts and alternating binary order. --configs 4/2 or --configs 4/3 selects mixed precision without a deadline cut. Each sample generates eight tokens to check the decode handoff; it does not measure complete answers or decode throughput.

Prepare the selected low-bit stores first. Any store build invalidates a sample. Reports retain hashes, source diff, prompts, telemetry, output, and power readings.

Results

The benchmark suite writes to results/<UTC timestamp>/, or the path given by --output. Smoke reports use results/*-smoke.json. New results are ignored by Git; retain reviewed runs with git add -f results/<name>.

Keep each run directory intact so relative output and image links resolve.

Baseline: September 9, 2026

This baseline contains 80 complete samples from a 32 GB M4 MacBook Air at revision 93c514f, using Sawfwair/Qwen3.8-Flash-Next-MLX-4bit.

• Timings, pelicans, and full answers
• Raw data and hashes
• Benchmark method

Setup

All runs used fresh processes, two adaptive drafts, an adaptive expert pool, and 8,192-token context. No CPU checks, prefix caching, or developer overrides were enabled. Memory was 20.98 GB after prefill scratch release.

Batched prefill used Q4 in every mode. The mixed setting also used --cut-weak 0.08, so its result measures both precision and a timing-dependent cut. All 18 workload/precision groups without the cut repeated byte-identically.

Exclusions

• The first mixed run built the Q2 store during its 75.08-second load. It was saved under setup_runs and excluded from medians. Prefill and decode were timed separately at 5.98 and 20.77 seconds.
• A battery transition invalidated r3-code-misses-2bit. It was retried on AC; the failed sample remains in previous_attempts.
• Capped answers were excluded. Final caps were 7,168 for LRU, reasoning, and pelicans. Earlier attempts and arguments remain in the report.

Interpretation

Compare completion time and answer length alongside token rates. Outputs are unedited and were not graded for correctness. The figures apply to the saved executable; its hashes are in the report. That executable and its development history are not distributed here. Local paths were replaced with placeholders.

Cherenkov benchmark

Source commit: 93c514f47e0f6c234aea47c11ccdf0c847f99dac

Run observations.

Pelicans

These are unedited model outputs from the benchmark.

4-bit	4-bit / 2-bit misses + cut

3-bit	2-bit

Answers

Low-bit prefill comparison

Timings and output checks cover 36 valid samples comparing Q4 prefill with per-expert low-bit prefill. Rates exclude loading and conversion.

Setup

Each sample ran on AC in a fresh process with an 8,192-token context, an adaptive pool, and two drafts. Prefix caching and deadline cuts were disabled. Each sample generated eight tokens to check the decode handoff. Binary order alternated between rounds; no stores were built during measurement. Memory was 20.98 GB after prefill scratch release.

Results

Q4 output matched in all six pairs. Q4 computation was unchanged, so its rate variation is a control. Low-bit prefill changes output. Long-prompt Q2 gains varied from 1.0% to 15.9% across the two pairs; that result is less certain than the short-prompt gains.

report.json contains hashes, timings, power readings, and source metadata. Archived executable paths refer to the measurement machine.

Low-bit prefill comparison

The report contains 36 valid samples. The medians exclude loading and conversion.

Output checks

• Round 1, 121 tokens: the Q4 outputs are identical.
• Round 1, 521 tokens: the Q4 outputs are identical.
• Round 1, 3221 tokens: the Q4 outputs are identical.
• Round 2, 121 tokens: the Q4 outputs are identical.
• Round 2, 521 tokens: the Q4 outputs are identical.
• Round 2, 3221 tokens: the Q4 outputs are identical.

Binary hashes, timings, source and power readings are in report.json.

Run observations.

Mixed-precision prefill comparison

Timings cover 12 valid samples comparing Q4 prefill with Q4 residents and Q2 misses. Rates exclude loading and conversion.

Each sample ran on AC in a fresh process with an 8,192-token context, an adaptive pool, and two drafts. Prefix caching and deadline cuts were disabled. Each sample generated eight tokens to check the decode handoff. Binary order reversed in round two. Q4/Q3 was not measured. Memory was 20.98 GB after prefill scratch release.

The binaries match the uniform comparison, whose report records the implementation revision. Low-bit prefill changes output. report.json contains hashes, timings, and power readings.

Low-bit prefill comparison

The report contains 12 valid samples. The medians exclude loading and conversion.

Binary hashes, timings, source and power readings are in report.json.

Run observations.