Expand description
Server startup and shared job ownership. One worker owns the loaded GPU.
Modules§
- failure 🔒
- HTTP status for expected lifecycle failures; validation defaults to 400.
- http 🔒
- Bounded HTTP input and JSON/SSE wire framing.
- output 🔒
- Slow clients consume bounded writer queues, never the GPU worker thread.
- pacer 🔒
- Wall-time pacing within the prefill capacity reserved at load.
- registry 🔒
- Bounded request registration and cancellation, independent of the GPU worker.
- request 🔒
- Completion options, chat rendering, tokenization and context policy.
- response 🔒
- Endpoint-specific JSON and SSE framing, independent of token generation.
- routes 🔒
- HTTP endpoints and admission into the generation queue.
- sessions 🔒
- Bounded conversation history, committed at the final-response publication boundary.
- stats 🔒
- Generation work, shared by active-request and committed-session snapshots.
- tool_
call 🔒 - Qwen tool-call output parsing.
- worker 🔒
- Round-robin GPU scheduling. Each active request owns its decoding state.
Structs§
Enums§
- ApiKind 🔒
Constants§
- MODEL 🔒