SW/HW Co-optimization Technique for LLMs — Half 2 (Software program) | by Liz Li | Jan, 2024


Thank you for reading this post, don't forget to subscribe!

SW is consuming the world. SW panorama of LLMs? What are the rising libraries/SW frameworks to enhance LLM efficiency?

Liz Li

Towards Data Science

With a continuing inflow of latest LLM fashions and options (try the hugging face LLM leaderboard), software program instruments and libraries are being launched at an accelerated fee. This fast development can also be sparking quite a few improvements in AI {hardware}. When optimizing LLMs from a system perspective, it’s essential to know that whereas ongoing analysis emerges day by day from main corporations and analysis institutes similar to Meta, Google, OpenAI, Nvidia, Stanford, and others, the software program stack/libraries can’t straight translate all the things into {hardware} for execution instantly. Solely a selective small set of software program options could be supported that requires a number of months (~6 months)of improvement for manufacturing. If these options must be supported in AI {hardware} accelerator, it calls for a good longer improvement cycle (2–4 years) in case of any architectural change. Addressing this discrepancy between software program and {hardware} optimization for LLMs poses a big problem, one which we purpose to sort out on this collection of posts!

Picture by Creator

Rising software program instruments and libraries cater to each LLM coaching and inferencing. On this publish, our focus will likely be particularly on LLM deployment and an in-depth exploration of how these instruments improve LLM efficiency. In an upcoming publish, we’ll delve into LLM coaching software program like deepspeed, Fairscale, colossal-AI, and extra.

Beforehand, I mentioned enhancements to LLM fashions and highlighted new analysis options. You possibly can refresh your reminiscence by revisiting that dialogue under

LLMs, designed as specialised area AI fashions, depend on the standard AI stack to transform fashions into machine code for execution on AI {hardware}. Totally different {hardware} corporations supply their respective software program stacks to facilitate AI inference. Under, I’ll showcase three distinguished {hardware} distributors…



Leave a Reply

Your email address will not be published. Required fields are marked *