GIPHY Goes to ESP32

July 14, 2021 | Lizzie Prader, Software Engineer

The latest addition to the Moddable SDK is a delightfully silly example app to display animated GIFs from the popular GIPHY web site. The app lets the user enter a search term, uses GIPHY's REST API to search for a GIF, downloads it, and then displays the GIF onscreen in a continuous loop.

Despite the silliness, the app is surprisingly complex. Doing the same thing in a web browser is easy. But, doing this on an embedded MCU like the ESP32 is unprecedented. It is great example of the benefits of using the Moddable SDK on inexpensive hardware like the ESP32.

This blog post explains the many technologies needed to make GIPHY work on an ESP32. It describes the technical challenges that need to be overcome to make this work and how the app uses modules in the Moddable SDK to overcome them. The post also walks through some of the app's source code to highlight the some of the JavaScript APIs essential to making this work.

The full source code for the app is available here.

Why is this interesting?

This project is interesting for a few reasons--not just because GIFs are fun (although that's true too).

First, GIFs can be practical. Modern user interfaces often feature animations, and GIFs allow animations to be created on a computer using and rendering techniques that are impossible on an embedded device. The resulting GIF can then be rendered on a device with very little code. GIFs are usually losslessly encoded so the fidelity is perfect, and the files are compressed so they save space. All of these characteristics make using GIFs great for adding visual sophistication to a product.

Second, this project features several technologies that are familiar to many people. Downloading and displaying GIFs on an ESP32 requires use of:

  • HTTP
  • TLS
  • REST APIs
  • JSON
  • Flash storage
  • Graphics and UI design
All of these tehcnologies are available together in the Moddable SDK through JavaScript APIs, making it straightforward to get them working together.

Finally, it's amazing that this works on a microcontroller like the ESP32. The challenges of making this all work may not be obvious. The next sections explain some of the technical details of this project and how the Moddable SDK is used to solve the technical challenges.

Search screen

The search screen is built using our expanding keyboard module. We designed this expanding keyboard to make text entry on small screens easier. You can read all about it in our Introducing an Expanding Keyboard for Small Screens blog post.

Using the GIPHY API

After the user enters a search query, the device sends an HTTPS request to GIPHY's Search Endpoint.

The host and path properties are set according to GIPHY's Search Endpoint API. In the path parameter,

  • API_KEY is a constant that stores a GIPHY API key
  • string is the string representation of the user's search query
  • offset is a number that specifies the starting position of the results GIPHY returns (this parameter is covered in more detail later in this section)
The response parameter specifies that the response should come as a string.

The port, Socket, and secure properties make the Request object use TLS so it is an HTTPS request, rather than an HTTP request.

let request = this.request = new Request({
    host: "api.giphy.com", 
    path: `/v1/gifs/search?api_key=${API_KEY}&q=${encodeURIComponent(string)}&limit=1&offset=${offset}&rating=g&lang=en`, 
    response: String,
    port: 443, 
    Socket: SecureSocket, 
    secure: {protocolVersion: 0x303} 
});
The app calls JSON.parse on the search result returned by GIPHY, passing in the string as the first argument and an array of keys as the second argument. The second argument here is a special feature of the XS JavaScript engine. It is an array listing the property names to parsed from the JSON. Property names not listed in the array are ignored. This is designed for situations where memory is especially tight. Using this feature can significantly reduce how much memory is used and how fast the JSON is parsed. This is critical here because GIPHY returns many properties which this app doesn't use, but would still take up memory if parsed. 
request.callback = function(message, value, etc) {
    if (Request.responseComplete === message) {
        let response = JSON.parse(value, keys);
        ...
The response variable at this point looks something like this: 
{
    "data": [
        {
            "url": "https://giphy.com/gifs/nehumanesociety-funny-dog-3o7527pa7qs9kCG78A",
            "username": "nehumanesociety",
            "title": "Confused What Is It GIF by Nebraska Humane Society",
            "images": {
                "fixed_width": {
                    "size": "539105",
                    "url": "https://media2.giphy.com/media/3o7527pa7qs9kCG78A/200w.gif?cid=d26008d1sz19nx8mpqkr41ri6ms7ym8d8c4vfvhcn2w4qz9c&rid=200w.gif&ct=g"
                }
            },
            "user": {
                "username": "nehumanesociety",
                "display_name": "Nebraska Humane Society"
            }
        }
    ],
    "meta": {
        "status": 200
    }
}
The images object typically provides more than one rendition of each GIF; apps use the rendition that works best for their purpose. The fixed_width and fixed_height renditions are recommended for preview grids like the one on GIPHY's homepage (shown in the image below); these renditions have a width of 200 or a height of 200, respectively. This app uses the fixed_width rendition because it fits well on the 240x320 screen of Moddable Two.

The GIF is downloaded to flash storage (more on this later). Before starting the download, the app checks whether the GIF will fit into the available flash space. If it's too big, it sends another request to GIPHY's Search Endpoint. For this request, it increments the offset by 1 so it get the next search result.

        ...
        let item = response.data[0];
        if (Number(item.images.fixed_width.size) > 2090000) { // Don't try to download images that are too big to fit into flash
            application.defer("searchGiphy", string, offset+1);
            ...
Once the app finds a GIF that fits into the available flash space, it triggers the downloadGif event, passing in the URL of the GIF and the title property from the response data (which contains the title and author of the GIF). 
        ...
        } else 
            application.defer("downloadGif", item.images.fixed_width.url, item.title);
    }
}

Writing the image data to flash

When the downloadGif event is triggered, the app makes another HTTPS request to download the image data. The image data comes in from GIPHY in chunks and, as mentioned earlier, is stored in flash. The reason GIFs are stored in flash is that the ESP32 has relatively little RAM (520 KB) compared to flash (4 MB); since GIFs tend to be relatively large files (2 MB is common), it is impossible to store the data of most GIFs in RAM on an ESP32.

Flash memory is divided into partitions. For example, one partition contains your project’s code, another the preference data, and another the storage for the file system. The ESP32 allows you to define how your flash memory is divided using a partition table as described in the ESP32 Partition Tables documentation. Each partition is identified by a name; you can see the names of the partitions for this app in the leftmost column of partitions.csv.

# Name,   Type, SubType, Offset,  Size, Flags
nvs,      data, nvs,     0x9000,  0x6000,
phy_init, data, phy,     0xf000,  0x1000,
factory,  app,  factory, 0x10000, 0x1c9000,
xs,       0x40, 1,       0x1d9000, 0x227000,
The Flash class lets you access the flash memory on your device. The following line of code instantiates the Flash class to access the xs partition. 
let partition = new Flash("xs");
As chunks of data come in, the app erases blocks of flash that are about to be written, then writes the data. You have to erase before writing because writing sets only the 0 bits. 
        request.callback = function(message, value, etc) {
            switch (message) {
                ...
                case Request.responseFragment: {
                    const data = this.read(ArrayBuffer);
                    const byteLength = data.byteLength;
                    while ((offset + byteLength) > blockIndex * blockSize) {
                        partition.erase(blockIndex);
                        blockIndex++;
                    }
                    partition.write(offset, byteLength, data);
                    offset += byteLength;
                    ...
You could erase the entire flash partition before sending the HTTPS request instead of erasing as you go. However, erasing such a large partition takes a long time (more than several seconds) and is unnecessary when the GIF being downloaded won't take up the entire partition. It's much more efficient and user-friendly to do it as you go.

Displaying the GIF

The user interface is implemented with the Piu user interface framework. This app uses a new Piu class, GIFImage, to display GIFs.

The GIFImage class is built with the Commodetto GIF decoder. The ReadGIF class provides access to the decoder. This app also uses the ReadGIF class directly.

The next sections explain how the ReadGIF and GifImage classes are used.

ReadGIF class

As described earlier, GIF data is stored in flash. This is possible because of how the Commodetto GIF decoder is implemented. Most GIF implementations for microcontrollers decode directly to the frame buffer to minimize memory use. The Commodetto GIF decoder decodes to a memory buffer. This increases the memory requirements, but it is the only way to accurately decode the full range of GIF files. Decoding directly to the frame buffer sometimes causes flickering or artifacts; decoding to a memory buffer avoids this problem.

The Commodetto GIF decoder has other useful features. One feature that is particularly useful for this app is the ready property of the reader instance. The ready property is true if at least one frame of the GIF is available to draw. While downloading images, the app checks reader.ready to determine when the reader is able to return the first frame. When the first frame is available, it triggers the showFirstFrame event so the first frame is drawn as soon as possible.

ready = reader.ready;
if (!ready) break;
application.first.defer("showFirstFrame");

GifImage class

The code that uses the GifImage class to display GIFs is in the screens module. The GifScreen template defines the screen that displays GIFs.

When a GifScreen is first instantiated, it contains four content objects:

  • A Container object with a back arrow
  • A Container object with an empty progress bar
  • A Label object that says "Loading..."
  • A Content object that shows the GIPHY logo
const GifScreen = Container.template($ => ({
    top: 0, bottom: 0, left: 0, right: 0, 
    Skin: BackgroundSkin, Style: TitleStyle,
    contents: [
        Container($, {
            anchor: "HEADER", height: 50, top: 0, left: 12, right: 12,
            contents: [
                Content($, {
                    left: 0, top: 14, Skin: BackArrowSkin
                })
            ]
        }),
        Container($, {
            anchor: "PROGRESS_BAR", left: 20, top: 56, height: 4, width: 200, Skin: ProgressBarSkin,
            contents: [
                Content($, {
                    left: 0, height: 4, width: 0, Skin: ProgressBarSkin, state: 1
                })
            ]       
        }),
        Label($, {
            anchor: "LOADING", Style: KeyboardStyle, state: 1, string: "Loading..."
        }),
        Content($, {
            anchor: "FOOTER", left: 25, top: 286, Skin: GiphyLogoSkin
        })
    ],
    active: true, Behavior: GifScreenBehavior
}))
The GifScreenBehavior has functions that are triggered by events during the download process. For example, every time a chunk of data is written to flash in main.js, the onUpdateProgress event is triggered: 
application.distribute("onUpdateProgress", offset/this.length);
This causes the onUpdateProgress function in GifScreenBehavior to be called, which updates the progress bar. 
onUpdateProgress(container, progress) {
    let background = this.data["PROGRESS_BAR"];
    let filler = background.first;
    filler.width = background.width * progress;
}
As you saw in the last section, the showFirstFrame event is triggered when the first frame of the GIF is downloaded. This calls the showFirstFrame function of GifScreenBehavior, which replaces the label that says "Loading..." with an instance of the GifImage class. The GifImage class constructor accepts a buffer property in its instantiating data; here it points to the memory buffer where the GIF data is stored (the xs partition in flash). 
showFirstFrame(container) {
    container.remove(this.data["LOADING"]);
    delete this.data["LOADING"]
    application.purge();
    let GIF = new GIFImage(this.data, { 
        anchor: "GIF", top: 65,
        buffer: new Flash("xs").map(), 
        Behavior:GIFImageBehavior 
    });
    container.insert(GIF, this.data["FOOTER"]);
}
Finally, when the GIF is fully downloaded, the onGifDownloaded event is triggered. The GifScreen responds to this event by removing the progress bar, hiding the header and footer, and starting the GIF animation. 
onGifDownloaded(container) {
    this.downloading = false;
    container.remove(this.data["PROGRESS_BAR"]);
    delete this.data["PROGRESS_BAR"]
    this.showControls = false;
    container.time = 0;
    container.duration = 200;
    container.start();
    this.data["GIF"].delegate("animate");
}

Conclusion

Downloading and displaying animated GIFs on an ESP32 is a surprisingly difficult task with a number of technical challenges. The Moddable SDK provides all the necessary tools to make it seem simple. The existing network and file modules make it straightforward to download GIF data; the Piu user interface framework and new `GIFImage` class make it straightforward to display the GIFs. And, of course, all of these tools aren't just great for silly apps like the `giphy` app--they're also great for consumer IoT products that want to get the most out of inexpensive hardware. via GIPHY