Driven by a desire for privacy, customization, and lower costs, there’s growing interest in AI models which can be run on local hardware. Few of them go as far as [Tim], though, who built an image generation diffusion model which can run on an RP2350 microcontroller.
As might be expected, its capabilities are limited. The resolution is 128×128, it only generates images of human faces, and it takes about twenty seconds per image – still impressive for such limited hardware. It runs on a Waveshare RP2350 development board, and it can output the generated image over USB or display it with the aid of a VGA adapter board.
The generative model doesn’t directly create an image. Rather, it generates a distribution in a latent space, which a variational auto-encoder’s decoder component translates into an image. The auto-encoder was trained in two parts: an encoder which transforms an image into a latent-space distribution, and a decoder to transform that distribution back to an image; once this was trained, only the decoder was used.
The generative portion of the model uses a latent flow diffusion transformer; this takes in noise to start with, then iteratively predicts changes which bring it toward the desired image. It can also take in a output class, which guides the generator’s direction (toward a smiling face, for example). [Tim] trained two models, one larger and one faster, and quantized the weights for both to 8-bit integers. Both models, along with the inference program, then fit into 4 MB of flash memory.
For such a small model, the results are remarkably good; they don’t look quite natural, but they’re quite recognizable. For more on how diffusion image generators work, check out our article on Stable Diffusion.
You must be logged in to post a comment Login