Choose the right deployment type
See Core concepts for details. Use Serverless in non-production environments and configure worker timeout to shut down idle instances.
Use quantization or lower precision
Quantization and lower-precision formats reduce memory usage by representing weights and activations with fewer bits. This lowers infrastructure cost and often improves throughput. Supported methods include INT4 and INT8 quantization, as well as FP8, BF16, and EETQ where available.Memory and cost benefits
- 4-bit quantization (INT4): ~75% less memory than FP16. A 33B model can run in ~24 GB VRAM; a 65B model in ~46 GB.
- 8-bit quantization or FP8 (INT8/FP8): ~50% memory reduction with minimal quality impact; often 1.5×–2× faster inference.
Shorten the context window for LLMs
A shorter context window reduces VRAM needed to store activations. For example, reducing a 32k context to 16k typically cuts context memory roughly in half. Set the context length in your model’s deployment configuration to match your application’s actual needs.Tune autoscaling and worker timeout
- Raise scale up threshold if you prefer fewer, busier instances.
- Lower worker timeout to shut down idle instances faster on Serverless deployments.
- Use minimum instances = 0 (where supported) for workloads that tolerate instance load time.
Monitor usage
Track per-model cost, token usage, and instance count on the Analytics tab. See Manage models and Billing.Billing safeguards
- Enable auto-pay so deployments are not interrupted when credits run low.
- Review estimated costs in the deployment UI before confirming.
- Scope API keys to specific models to limit blast radius.