paged-llama

A Llama-family inference engine written from scratch in C++: safetensors loader, byte-level BPE, the transformer forward pass, a paged KV cache and continuous batching, serving the OpenAI API shape. This box is two ARM cores with no GPU, so expect tens of tokens per second, not hundreds.

max tokens

first token –tokens –rate – tok/sfinish –

The same request, from a shell: