A model-fit engineering notebook
A local model that loads successfully has not yet passed a workload test. The useful sizing envelope combines the exact model build with quantisation, working context, active sequences, runtime behaviour and operating headroom.
Use this page when model fit is the main question. Start with the smallest model that meets the accepted task, then prove memory and service behaviour.
LLM model-fit notebook
A local LLM deployment is shaped by more than parameter count. The model build, quantisation, context, KV cache, active sequences and runtime must fit together with operational headroom.
Record the exact model build and precision rather than relying on a family name or parameter headline.
Test the working context and KV cache required by the real documents, prompts and output lengths.
Measure overlapping requests. A model that fits for one user can fail when several sequences stay active.
Decide whether the service needs one GPU, independent workers or supported model parallelism.
Leave room for monitoring, updates and demand variation, then retain the tested configuration.
Runtime starting points
| Runtime | Useful starting fit | Evidence to retain |
|---|---|---|
| Ollama | Approachable managed model use and smaller team deployments | Pinned model, API behaviour, access boundary and representative task |
| vLLM | Higher-throughput LLM serving and OpenAI-compatible endpoints | Concurrency, latency, throughput, cache use and failure behaviour |
| llama.cpp | Flexible quantised inference across varied hardware | Exact build, offload, context, tokens per second and output quality |
Model weights consume VRAM, but so do context, KV cache, batch size and concurrent sequences. Runtime overhead and quantisation format change the result again.
A useful test set contains representative questions, source documents and expected answers. Quality, latency and resource use are measured together.
One high-memory GPU is the simplest execution target. Several GPUs can serve independent models or users efficiently. A larger model can be split only where the runtime and topology support it.
The website therefore states per-GPU and aggregate VRAM, and never markets aggregate memory as universally pooled.
Ollama is approachable for managed model use and smaller deployments. vLLM is designed for higher-throughput serving. llama.cpp gives flexible quantised inference across many platforms.
The chosen profile is pinned and documented. A name in a package list is not evidence that every workload will behave the same way.
Local open models can be excellent for retrieval, summarisation, extraction, coding and structured workflows. They may not match the strongest hosted models on every reasoning or multimodal task.
A hybrid route can preserve a deliberate frontier-model path while keeping routine or sensitive work local.
Questions answered
It depends on model weights, quantisation, context, batch size, concurrency and runtime. Start with a measured workload and use the VRAM guide rather than a single parameter-to-memory rule.
Some runtimes can divide supported models across GPUs, but the result depends on software and interconnect. Aggregate VRAM is not a universal pool.
No. SaaS is usually the better-value choice for a small team with light or irregular use. Owned capacity becomes more credible when privacy, offline operation, many shared users or sustained workloads have independent value. We show the comparison rather than forcing the server answer.
Primary-source register
Reviewed 26 July 2026. These links support the dated statements on this page; they do not replace legal, tax, security or professional advice.
Continue the decision
Resolve the model-fit question
The useful output is a testable memory and runtime profile, not a universal claim about a parameter count.