Skip to main content

prefill_engine

Function prefill_engine 

Source
fn prefill_engine(
    gpu: &mut Gpu<'_>,
    cpu: Option<&CpuModel<'_>>,
    cpu_state: Option<&mut State>,
    ids: &[u32],
    n_draft: usize,
    pf_chunk: usize,
    argmax_lines: Option<&mut Vec<String>>,
) -> Result<PrefillResume>
Expand description

Stream equal-sized prompt chunks so a short tail does not reread the whole store.