A weights-only calculation is a useful first filter. A deployment decision needs context length, concurrency and a measured memory budget.
· Arvica Cloud · Analysis & buying guide
For a hypothetical model with exactly 70 billion parameters, storing every weight at two bytes takes 140 billion bytes: 140 GB in decimal units, approximately 130.4 GiB. At an ideal packed four bits per weight, the raw payload is 35 GB, approximately 32.6 GiB. These are arithmetic estimates for weights only, not required device capacity. Quantisation metadata, non-quantised tensors and the particular checkpoint format change the actual allocation. Do not describe a card as suitable solely because its advertised memory exceeds the smaller number.
vLLM documents how KV-cache pressure and scheduling settings affect serving; insufficient cache can cause requests to be preempted and recomputed. [1] Our procurement recommendation is to test the busiest realistic request mix after loading the model. Include long prompts, generated output and simultaneous users. Reserve a separate line in the budget for runtime allocations. A single short demonstration request says little about a document assistant receiving several large documents together.
Choose a short and a long prompt length from your own traffic, then test low and expected peak concurrency at each length. Keep model revision, output limit and precision fixed. Record startup peak memory, steady-state memory, request failures, time to first token and output speed. Repeat after changing one setting. This is a proposed test plan, not an Arvica benchmark: no configuration or speed is certified by the arithmetic above.
Send the exact model repository and revision, quantisation format, longest supported input, expected concurrent requests and response-time target. If you already have an out-of-memory log, include a redacted excerpt with the software version. Ask for an exact configuration and a trial acceptance criterion, rather than asking only for a GPU model. If quality changes under quantisation, compare business-task accuracy before accepting the cheaper memory footprint.
Reviewed: 2026-09-11
No charge before quote acceptance; the infrastructure provider is disclosed before activation.
Build a representative inference test with model, precision, context, concurrency and quality targets. Connect the measured result to a transparent GPU budget.
Download an editable worksheet to agree scope, compare configurations and decide whether to stop, adjust or scale. An evaluation method, not a customer case study.
