A shared service, not just loaded weights
An inference server runs trained models for people or applications, but the hardware is only one dependency. The request boundary, queue, worker profile, telemetry and recovery route determine whether it is a usable service.
Use this page when the service-level question is primary: how requests enter, wait, complete, fail and recover under representative demand.
Inference service control plane
Inference hardware becomes a shared service only when requests are authenticated, queued, assigned to a versioned model worker, observed and recovered against an agreed service target.
Authenticate the user or application and reject unapproved routes.
Apply limits, priority, timeouts and model-routing rules.
Serve the pinned model, runtime, context and batching profile.
Return the output while recording errors, latency and permitted logs.
Restart, roll back or use an approved fallback when the service fails.
Acceptance board
Record the model, runtime, hardware, context, batch profile and test date beside every result. A number without its conditions is not a buying claim.
| Measure | Decision question | Evidence record |
|---|---|---|
| Latency | How quickly must useful output begin and finish? | Median and tail results under stated conditions |
| Throughput | How much accepted work must complete in a period? | Requests, tokens or jobs with the model and batch profile |
| Concurrency | How many active sequences overlap at the real peak? | Queue time, cache use and rejection behaviour |
| Recovery | What happens after a worker, runtime or model failure? | Health check, restart, rollback and approved fallback |
Operating service brief
A developer can load a model on a workstation. A production-style service needs an API, identity, queues, timeouts, monitoring, version control and a recovery process.
The acceptance test covers representative prompts, context, concurrency and latency - not a best-case single request.
Batching can improve total throughput while increasing individual response latency. Long contexts consume more memory. Several users can exhaust KV cache even when the base model fits.
The right answer may be one large endpoint, several independent workers or a routing layer across different models.
Internal applications can use an OpenAI-compatible local API while staff use a reviewed browser interface. Access must be authenticated and scoped to the customer network.
API compatibility does not mean identical model behaviour. Applications still need model-specific evaluation and error handling.
This proposition is not hourly hosted GPU rental. The customer buys the configured appliance and chooses where it runs.
Colocation may still be sensible for high-power systems. Ownership and physical location are separate decisions.
Questions answered
A server that loads trained AI models and serves outputs to users or applications. It is designed around response latency, throughput, concurrency and operational availability rather than model training.
Inference machine is a narrower phrase for hardware intended primarily to run trained models. AI server is broader and can include training, rendering, data processing and other GPU workloads.
There is no responsible universal number. It depends on model, context, output length, quantisation, latency target and traffic pattern. The exact workload needs a concurrency test.
Primary-source register
Reviewed 26 July 2026. These links support the dated statements on this page; they do not replace legal, tax, security or professional advice.
Continue the decision
Resolve the service question
Record latency, throughput, peak concurrency, access and recovery so the exact server can be tested against them.