GPU inference optimisation

When this helps

For teams running GPU models with queuing requests, variable first-token or tail latency, or memory limits under changing traffic. Also for teams needing to understand batching, throughput and response time trade-offs before changing serving configuration.

Establish a serving baseline

Define the model, GPU resources, request shapes and serving configuration. Measure queue time, time to first token (TTFT), tail latency, throughput and errors on representative inputs rather than relying on a single average.

Separate prefill (prompt processing) from decode (token generation) where measurements allow. Their resource use and delays differ, helping locate whether waiting comes from queued work, prompt processing or generation.

Review queueing and batching behaviour

Observe how requests wait and how batching changes as traffic varies. Batching affects how much work the GPU handles together, but its impact on response time depends on request lengths, arrival patterns and serving setup.

Use controlled load conditions that resemble the target workload. Compare throughput with first-token and tail latency, and record errors or resource pressure so throughput changes include their service impact.

Check memory use and model settings

Review GPU memory use, including space for active request context such as the key-value (KV) cache. Request length and concurrency affect how much context fits, so operating conditions matter when evaluating capacity.

Quantisation and other serving settings may change memory requirements and output behaviour. Assess options against the model's quality needs and workload measurements rather than assuming consistent behaviour across hardware.

Compare changes under controlled load

Select experiments based on observed bottlenecks and compare them using consistent inputs and load conditions. Track queueing, TTFT, tail latency, throughput, memory pressure and errors together to see if changes serve more requests effectively on existing hardware.

Document sustained request rate, test duration and response-time distribution for each configuration, including quality changes or errors. The handover records these operating conditions, the configuration to deploy, monitoring and a rollback plan.

What you receive

• Workload and configuration baseline: request patterns, queueing, TTFT, tail latency and throughput.
• Review of prefill and decode behaviour, batching, concurrency and GPU memory use where measurable.
• Controlled comparisons of selected serving or quantisation settings, including quality and error checks.
• Agreed configuration changes with monitoring guidance and a rollback plan.

Common questions