28.9M LLM Squeezed onto ESP32-S3: A Surprisingly Capable AI Judge

A team of developers has successfully fitted a large language model (LLM) with 28.9 million parameters onto an ESP32-S3 microcontroller, demonstrating the potential for AI to run on even the smallest devices. This achievement is particularly noteworthy given the tiny size and limited RAM of the ESP32-S3.

The LLM in question can tell coherent stories and keep track of variables, showcasing its ability to perform complex tasks despite being confined to a relatively small space. For context, this model has roughly a quarter of the parameters found in OpenAI’s first ChatGPT version, which had 117 million parameters.

So how did they manage to pack so many parameters onto such a tiny device? The key lies in an innovative technique called Gemma’s Per-Layer Embeddings. This approach recognizes that most language model parameters are stored in a large embedding table, rather than being computed on the fly.

The solution was to store this 25 million-row table in flash memory and read only a few rows at a time when needed. By doing so, they were able to free up fast RAM for more critical tasks, requiring only around 560 kilobytes of dense core space.

This clever use of memory-mapped XIP (execute-in-place) allows the model to run efficiently on-chip, processing approximately nine tokens per second. While this may not be as fast as larger AI models, it’s a remarkable feat considering the ESP32-S3’s limited resources.

The project is now available for download from the ESP32-AI GitHub page, allowing developers and enthusiasts to experiment with their own LLMs on small devices.