Use quantization or lower precision
Quantization and lower-precision formats reduce memory usage by representing weights and activations with fewer bits. This lowers infrastructure cost and often improves throughput. Supported methods include INT4 and INT8 quantization, as well as FP8, BF16, and EETQ where available.Memory and cost benefits
- 4-bit quantization (INT4): ~75% less memory than FP16. A 33B model can run in ~24 GB VRAM; a 65B model in ~46 GB.
- 8-bit quantization or FP8 (INT8/FP8): ~50% memory reduction with minimal quality impact; often 1.5×–2× faster inference.
Shorten the context window for LLMs
A shorter context window reduces VRAM needed to store activations. For example, reducing a 32k context to 16k typically cuts context memory roughly in half. Set the context length in your model’s deployment configuration to match your application’s actual needs.Tune autoscaling and worker timeout
- Raise scale up threshold if you prefer fewer, busier instances.
- Lower worker timeout to shut down the last idle instance sooner.
- Use minimum instances = 0 for workloads that tolerate instance load time. Keep minimum instances above 0 when you need an instance already loaded.
Monitor usage
Track per-model cost, token usage, and workers on the Usage tab. See Manage models, account Usage, and Billing.Billing safeguards
- Enable auto-pay so deployments are not interrupted when credits run low.
- Review estimated costs in the deployment UI before confirming.
- Scope API keys to specific models to limit blast radius.