Choose the right readiness level
See Core concepts for details. Downgrade readiness in non-production environments and use worker timeout to shut down idle instances.
Use quantization or lower precision
Quantization and lower-precision formats reduce memory usage by representing weights and activations with fewer bits. This lowers infrastructure cost and often improves throughput. Supported methods include INT4 and INT8 quantization, as well as FP8, BF16, and EETQ where available.Memory and cost benefits
- 4-bit quantization (INT4): ~75% less memory than FP16. A 33B model can run in ~24 GB VRAM; a 65B model in ~46 GB.
- 8-bit quantization or FP8 (INT8/FP8): ~50% memory reduction with minimal quality impact; often 1.5×–2× faster inference.
Shorten the context window for LLMs
A shorter context window reduces VRAM needed to store activations. For example, reducing a 32k context to 16k typically cuts context memory roughly in half. Set the context length in your model’s deployment configuration to match your application’s actual needs.Tune autoscaling and worker timeout
- Raise scale up threshold if you prefer fewer, busier instances.
- Lower worker timeout to shut down idle instances faster on Super fast and Cold start deployments.
- Use minimum instances = 0 (where supported) for workloads that tolerate cold starts.
Monitor usage
Track per-model cost, token usage, and instance count on the Analytics tab. See Manage models and Billing.Billing safeguards
- Enable auto-pay so deployments are not interrupted when credits run low.
- Review estimated costs in the deployment UI before confirming.
- Scope API keys to specific models to limit blast radius.