AMD Introduces AMD-135M: A Breakthrough in Small Language Models

In a big improvement throughout the synthetic intelligence sector, AMD has introduced the discharge of its first small language mannequin (SLM), AMD-135M. This new mannequin goals to supply specialised capabilities whereas addressing a number of the limitations confronted by giant language fashions (LLMs) akin to GPT-4 and Llama, in response to AMD.com.

AMD-135M: First AMD Small Language Mannequin

The AMD-135M, a part of the Llama household, is AMD’s pioneering effort within the SLM area. The mannequin was skilled from scratch utilizing AMD Intuition™ MI250 accelerators and 670 billion tokens. The coaching course of resulted in two distinct fashions: AMD-Llama-135M and AMD-Llama-135M-code. The previous underwent pretraining with normal knowledge, whereas the latter was fine-tuned with a further 20 billion tokens particularly for code knowledge.

Pretraining: AMD-Llama-135M was skilled over six days utilizing 4 MI250 nodes. The code-focused variant, AMD-Llama-135M-code, required a further 4 days for fine-tuning.

All related coaching code, datasets, and mannequin weights are open-sourced, enabling builders to breed the mannequin and contribute to the coaching of different SLMs and LLMs.

Optimization with Speculative Decoding

One of many notable developments in AMD-135M is using speculative decoding. Conventional autoregressive approaches in giant language fashions usually endure from low reminiscence entry effectivity, as every ahead cross generates solely a single token. Speculative decoding addresses this by using a small draft mannequin to generate candidate tokens, that are then verified by a bigger goal mannequin. This methodology permits a number of tokens to be generated per ahead cross, considerably enhancing reminiscence entry effectivity and inference pace.

Inference Efficiency Acceleration

AMD has examined the efficiency of AMD-Llama-135M-code as a draft mannequin for CodeLlama-7b on varied {hardware} configurations, together with the MI250 accelerator and the Ryzen™ AI processor. The outcomes indicated a substantial speedup in inference efficiency when speculative decoding was employed. This enhancement establishes an end-to-end workflow for coaching and inferencing on chosen AMD platforms.

Subsequent Steps

By offering an open-source reference implementation, AMD goals to foster innovation throughout the AI neighborhood. The corporate encourages builders to discover and contribute to this new frontier in AI know-how.

For extra particulars on AMD-135M, go to the complete technical weblog on AMD.com.

Picture supply: Shutterstock