Nvidia’s banking on TensorRT to expand its generative AI dominance – The Verge

Nvidia announced that its adding support for its TensorRT-LLM SDK to Windows and models like Stable Diffusion as it aims to make large language models (LLMs) and related tools run faster. TensorRT speeds up inference, the process of going through pretrained information and calculating probabilities to come up with a result like a newly generated Stable Diffusion image. With this software, Nvidia wants to play a bigger part on that side of generative AI.

TensorRT-LLM breaks down LLMs like Metas Llama 2 and other AI models like Stability AIs Stable Diffusion to let them run faster on Nvidias H100 GPUs. The company said that by running LLMs through TensorRT-LLM, this acceleration significantly improves the experience for more sophisticated LLM use like writing and coding assistants.

This way Nvidia can not only provide the GPUs that train and run LLMs but also provide the software that allows models to run and work faster so users dont seek other ways to make generative AI cost-efficient. The company said TensorRT-LLM will be available publicly to anyone who wants to use or integrate it and can access the SDK on its site.

Nvidia already has a near monopoly on the powerful chips that train LLMs like GPT-4 and to train and run one, you typically need a lot of GPUs. Demand has skyrocketed for its H100 GPUs; estimated prices have reached $40,000 per chip. The company announced a newer version of its GPU, the GH200, coming next year. No wonder Nvidias revenues increased to $13.5 billion in the second quarter.

But the world of generative AI moves fast, and new methods to run LLMs without needing a lot of expensive GPUs have come out. Companies like Microsoft and AMD announced theyll make their own chips to lessen the reliance on Nvidia.

And companies have set their sights on the inference side of AI development. AMD plans to buy software company Nod.ai to help LLMs specifically run on AMD chips, while companies like SambaNova already offer services that make it easier to run models as well.

Nvidia, for now, remains the hardware leader in generative AI, but it already looks like its angling for a future where people dont have to depend on buying huge numbers of its GPUs.

Continued here:

Nvidia's banking on TensorRT to expand its generative AI dominance - The Verge