LoRA fine-tuning vs inference: do you need the same GPU?
Separate training memory from serving memory, then size a LoRA trial using sequence length, batch size and a realistic evaluation.
· Arvica Cloud · Analysis & buying guide
Can I fine-tune a model that already fits for inference?
Not necessarily. Training introduces allocations for activations, gradients and optimiser state, depending on the method and implementation. A successful inference request is not a training capacity test. Record the exact base model, method, precision, maximum sequence length and micro-batch size before choosing a rental configuration.
What does LoRA reduce?
LoRA trains low-rank adapter parameters while freezing the original model weights. Hugging Face explains that this reduces trainable parameters and associated training memory. [1] It does not remove the base model or every runtime allocation. QLoRA additionally uses a quantised base model; the actual memory requirement still depends on the implementation and workload. Avoid a universal claim that every model of a given parameter count fits a particular card.
How should I run a sizing trial?
Use a representative sample containing your longest expected sequences. Measure peak memory and step time after warm-up; include evaluation and checkpoint saving. If you change micro-batch size or gradient accumulation, verify both throughput and the intended effective batch. Estimate the full run from measured steps, then allow for evaluation, checkpointing and recovery. Validate task quality before scaling the rental.
What information helps Arvica size the job?
Include the base model, LoRA or QLoRA method, dataset size, sequence length, training steps or epochs, framework version, deadline and preferred region. For the later inference service, separately specify concurrency and latency. The training rental and the serving deployment may need different configurations and terms.
Build a representative inference test with model, precision, context, concurrency and quality targets. Connect the measured result to a transparent GPU budget.
Download an editable worksheet to agree scope, compare configurations and decide whether to stop, adjust or scale. An evaluation method, not a customer case study.
