NVIDIA Enhances TensorRT Model Optimizer v0.15 with Improved Inference Performance

NVIDIA has launched the newest v0.15 launch of the NVIDIA TensorRT Mannequin Optimizer, a cutting-edge quantization toolkit designed to reinforce mannequin optimization methods resembling quantization, sparsity, and pruning. This replace goals to scale back mannequin complexity and optimize the inference velocity of generative AI fashions, in line with NVIDIA Technical Weblog.

Cache Diffusion

The brand new model consists of assist for cache diffusion, constructing on the beforehand established 8-bit post-training quantization (PTQ) method. This function accelerates diffusion fashions at inference time by reusing cached outputs from earlier denoising steps. Strategies like DeepCache and block caching optimize inference velocity with out extra coaching. This mechanism leverages the temporal consistency of high-level options between consecutive denoising steps, making it suitable with fashions like DiT and UNet.

Builders can allow cache diffusion by utilizing a single ‘cachify’ occasion within the Mannequin Optimizer with the diffusion pipeline. As an example, enabling cache diffusion in a Steady Diffusion XL (SDXL) mannequin on an NVIDIA H100 Tensor Core GPU delivers a 1.67x speedup in photographs per second. This speedup additional will increase when FP8 can also be enabled.

Quantization-Conscious Coaching with NVIDIA NeMo

Quantization-aware coaching (QAT) simulates the results of quantization throughout neural community coaching to get better mannequin accuracy post-quantization. This course of includes computing scaling elements and incorporating simulated quantization loss into the fine-tuning course of. The Mannequin Optimizer makes use of customized CUDA kernels for simulated quantization, reaching decrease precision mannequin weights and activations for environment friendly {hardware} deployment.

Mannequin Optimizer v0.15 expands QAT integration assist to incorporate NVIDIA NeMo, an enterprise-grade platform for creating customized generative AI fashions. This primary-class assist for NeMo fashions permits customers to fine-tune fashions straight with the unique coaching pipeline. For extra particulars, see the QAT instance within the NeMo GitHub repository.

QLoRA Workflow

Quantized Low-Rank Adaptation (QLoRA) is a fine-tuning method that reduces reminiscence utilization and computational complexity throughout mannequin coaching. It combines quantization with Low-Rank Adaptation (LoRA), making giant language mannequin (LLM) fine-tuning extra accessible. Mannequin Optimizer now helps the QLoRA workflow with NVIDIA NeMo utilizing the NF4 information kind. For a Llama 13B mannequin on the Alpaca dataset, QLoRA can scale back peak reminiscence utilization by 29-51% whereas sustaining mannequin accuracy.

Expanded Assist for AI Fashions

The newest launch additionally expands assist for a wider suite of AI fashions, together with Stability.ai’s Steady Diffusion 3, Google’s RecurrentGemma, Microsoft’s Phi-3, Snowflake’s Arctic 2, and Databricks’ DBRX. For extra particulars, check with the instance scripts and assist matrix out there within the Mannequin Optimizer GitHub repository.

Get Began

NVIDIA TensorRT Mannequin Optimizer gives seamless integration with NVIDIA TensorRT-LLM and TensorRT for deployment. It’s out there for set up on PyPI as nvidia-modelopt. Go to the NVIDIA TensorRT Mannequin Optimizer GitHub web page for instance scripts and recipes for inference optimization. Complete documentation can also be out there.

Picture supply: Shutterstock