Cross-buliding containers with Qemu User Static and buildah

I recently decided I should be containerizing my services as best I can. My network doesn't really lend itself to container deployments, since there's a lot of one-off hardware going on, but it's at least possible to prepare releases on other hardware (my laptop) so I've been looking into cross-building.

I'm currently working on a container to run Plume, the blogging platform I've published this post on, on an aarch64 host! How? It's actually not too complicated.

1. Install Podman and Buildah

I'm sure many readers will have heard of docker, but I've recently found podman and buildah to be really nice tools!. If you're on elementary OS like me, or on any Ubuntu derivative it's a simple sudo apt install podman buildah. If you're on Fedora or RHEL, you may already have them! If not, try a dnf install podman buildah. If you're on Arch it's pacman -S podman buildah.

Podman has some advantages over Docker, like it's ability to run without root in many cases. We'll see that used in this tutorial a bit. buildah, the container build tool, is incredibly helpful. It allows you to provision a container you would a normal host.

2. Configure podman

By default, podman likely won't know how to communicate with dockerhub. We'll need it to be able to do this in order to install some required containers later. Create a file called /etc/containers/registries.conf if one doesn't already exist, and make sure that these lines are inside it:

[registries.search]
registries = ['docker.io']

If your system came with this file, so long as 'docker.io' is included in the registries list, you should be okay.

3. Install qemu-user-static

On the *buntus, it's an apt install qemu-user-static. On the RHELs, it's a dnf install qemu-user-static. On Arch, it's pacman -S qemu-arch-extra.

qemu-user-static is a collection of interpreters that allow qemu, a virtualization platform, to run systems from non-native architectures. This means that you'll now be able to virtualize ARM, MIPS, and other platforms.

4. Run the multiarch qemu-user-static build container

This will enable a few settings to allow running non-native containers. It will also be the only time we use podman with sudo.

$ sudo podman run --rm --privileged multiarch/qemu-user-static --reset -p yes

You're all set up!

"But wait!" You say, "Just because I have the tools installed doesn't mean I know how to use them!" Okay that's a fair point

5. Start building a fedora aarch64 container with buildah

So I mentioned installing buildah earlier, and now we'll see how it's used

$ container=$(buildah from multiarch/fedora:30-aarch64)

We'll start by creating a buildah container based on a 64bit ARM Fedora container, and store the result of this command in the container variable. This is an easy way to avoid learning what the container's name is. If we care, though, we can do this

$ echo "$container"
=> fedora-working-container

Alright, so we're building a container called fedora-working-container. if you close and re-open your terminal, you can find the container name with this command:

$ buildah ls

under the CONTAINER NAME heading, you'll see fedora-working-container

6. Run a command in the container

$ buildah run $container bash

Now you have a bash shell inside your aarch64 container! Since this is a fedora container, you'll have access to dnf to install packages. Since you have shell access, you can build this system up the same way you would a VM or a bare-metal setup, creating users and installing packages.

7. Save your container

Now that you've spent the time building this container up to do exactly what you need, you need to save it! Luckily, this is incredibly simple.

$ buildah commit $container plume-fedora

You can see that we've saved a new image called plume-fedora now,

$ buildah ls

8. Publish your image

You can log into dockerhub, or another registry with the buildah login command. Once you've gotten your credentials set up, you can push your images with buildah.

$ buildah push plume-fedora docker://hub.docker.io/asonix/plume-fedora

9. Run you image with Podman

Now that your image is public, you can run it on any aarch64 system with podman or docker!

$ podman run asonix/plume-fedora