Tiny AI-Generated Images from a Microcontroller
A developer has pushed the boundaries of what’s thought possible for large language models by running one on an extremely low-cost microcontroller. The LLM in question, with its massive 28.9 million parameters, runs entirely locally on an ESP32-S3 chip that costs just $8.
The project uses a trick borrowed from Google’s Gemma models called Per-Layer Embeddings to fit the model into the microcontroller’s memory. This technique stores approximately 25 million parameters in flash memory as a lookup table, which dramatically reduces memory requirements during inference and lets the full model occupy just 14.9MB.
According to the developer, this is the first known demonstration of Google’s Per-Layer Embeddings concept being adapted to hardware as constrained as a microcontroller. The ESP32-S3 chip has only 512KB of SRAM, 8MB of PSRAM, and 16MB of flash storage, but it manages to run the model smoothly.
Despite its limitations, the model can generate short, simple stories with reasonably coherent structure. It was trained on Microsoft’s TinyStories dataset and is not intended for tasks like answering questions or writing code. The project’s GitHub repository includes all necessary files – firmware, training scripts, quantization pipeline, wiring instructions, and experimental results.
The generated text is written directly to a small display connected to the board at roughly 9.5 tokens per second. This implementation significantly outperforms earlier projects that ran LLMs on microcontrollers with much smaller models containing around 260,000 parameters – it’s about 100 times larger and raises important questions about how such large models can be made to fit on hardware with limited memory.
The project’s success has far-reaching implications for the development of AI-generated content. While this particular implementation focuses solely on text generation – rather than creating images directly – its potential applications extend into other areas, including image generation and beyond.