Docker is a leaky abstraction

This blog post details how I finally understood that Docker is not the perfect abstraction I thought it was and how the hardware your docker image will run on still matters.

In my efforts to improve my Ops skills, I have taken to using Docker wherever possible. By exposing myself continually to it I can understand better how containerisation works, where it is useful and more importantly, where it is not useful along with the pitfalls involved.

In my mind, Docker was this great layer of abstraction that provided value in the form of vague keywords such as reproducible builds and isolated dependencies. Whilst I have definitely reaped those benefits, a recent experience of trying to move some software from a machine with a particular CPU architecture to another has made me realise that Docker (or any flavour of containerisation) does not insulate a developer’s code completely from the machine it runs on. Like any abstraction, it leaks.

For a particular project I am using AWS. Typically it feels over-engineered for my needs which are usually met by Hetzner, Digital Ocean or any smaller VPS provider. In this case, latency was important and the service we were interacting with also runs on AWS, so I wanted to deploy into the same AWS region (more on this project in another blogpost 😉).

I try to run my sideprojects on shoe string budgets. I believe that it helps encourage learning by doing and stimulates creativity over simply throwing money at the problem.

During the course of this project we deployed some code written in Python (rather than Go), and began experiencing CPU saturation on our AWS Lightsail VPS. This was a pressing issue as it was causing us to respond slowly, sometimes exceeding the short timeout provided to us (we have to respond within 500ms).

There were several solutions to solving CPU usage in this case, each with their own tradeoffs:

1. Moving to a cheaper cloud provider
2. Optimise the code to make it faster
3. Switch to a different offering within AWS

Moving to cheaper cloud provider

There are two main issues to this solution:

• The amount of vCPUs available to a VPS tend to scale slowly compared to other elements such as RAM or Disk space. For instance, Digital Ocean will only offer more CPU when going from $5 a month to $15 a month.
• Moving further away from AWS will introduce network latency, which counted towards our 500ms deadline. I could get 3 AMD vCPUs for around €8 a month from Hetzner, however they only offer datacentres in Europe. From my testing, the latency difference could be up to 100ms from us-west-2 to Hetzner’s Nuremberg Data Centre.

Optimise the code to make it faster

There was one main issue to this solution: we are still heavily in the experimenting and prototyping phase of the project. By spending time on optimising the code (whether the existing code or by rewriting it in a compiled language), we would be burning precious time which would be better spent on more prototyping.

Switch to a different offering within AWS

So far we had used AWS Lightsail to provision a VPS, since it was the offering most familiar to us. Lightsail however offers very little control of the kind of hardware you run on (kind of the point, you shouldn’t need to think about it!).

I had heard about AWS Graviton, which are AWS’ custom ARM processors. When I checked the price, it turned out I could get 2vCPUs on a t4g.nano EC2 instance for the same price as what I was paying for 0.5vCPU on Lightsail.

Moving to Graviton

Moving from Lightsail to EC2 was not particularly diffiicult, as I mostly worked through Terraform and had some prior experience playing around with the EC2 interface. Moving to a graviton instance however is where the learnings came from.

If you ever try to run something compiled for x86 (the CPU architecture of both my laptop, my CI server and our Lightsail instance) on an ARM machine, you’ll get cryptic error messages such as: exec user process caused: exec format error. This counts for Docker too.

The above error had already begun challenging my incorrect understanding of Docker - it was a Docker Image! I shouldn’t need to think about the machine! Well it turns out I did. I did not allow myself to be dissuaded by this and powered on, through several hours of trial and error.

It turns out that Docker has support for multi-CPU architectures on an image level using a command called buildx where one can setup a builder. Customising this builder allows users to compile across CPU architectures. In my case, on Ubuntu, I followed some guides which mentioned qemu and binfmt_misc to enable hardware emulation for buildx to operate. I admit I was mostly flying blindly at this stage. After multiple attempts I was able to manually build an arm64 image on my x86 laptop and verify this by moving the image to our graviton instance and run it as a container.

Great! But holy cow does it come at a cost. A none-too-special multi-stage Docker image packaging a single Go binary which typically took 60 seconds to build (which I personally considered quite slow), around 300s to build for arm64. These kind of long lead times really hampered our development cycle, slowing down our ability to prototype quickly.

The Solution

Not willing to take the slow deployment cycle due to the cost of emulation every time, I began looking into alternatives. One approach suggested to me by more than one person was to build the Docker image on the graviton instance itself.

Whilst a very simple and easy approach, it encourages treating the server as a pet over cattle, along with burning the same compute I wanted to improve. It was plausible that the act of building the Docker image would interfere the running of the code itself.

Another approach would be to either configure my CI/CD server (in my case I use Jenkins) to perform the hardware emulation (I’d still pay the emulation cost, but at least it is not tied to my laptop) or to source some ARM hardware and build the Docker Image on it. After a few more failed attempts, it resulted that the Raspberry Pi 3 or 4, running the 64 bit version of Raspbian (the default one provided on the imager is 32 bit) mirrors Graviton’s arm64 architecture.

Our final setup looks like this:

1. We write and test the code locally on our x86 laptops
2. When finished, we merge the code into the main branch
3. Jenkins (running on an x86 machine), polls the repository and picks up the new code
4. Jenkins Master hands off the job to a Raspberry Pi running the Jenkins Agent (by restricting the job to a specific label)
5. The Docker image is built for the arm64 architecture and deployed to the Graviton instance using Ansible

Caveats

Whilst this solution may feel rube-goldberg worthy, I want to reiterate that the aim is to learn over to provide the most cost-effective and bullet-proof solution.

This process also brings into question one of the main benefits of Docker: reproducible builds. A major advantage of Docker is being able to use the image built on the dev’s laptop into a test environment, then onto a staging environment and eventually production. If any of those steps involve a different CPU architecture however and you end up having to rebuild your image, are you still testing/running the same image? How can you be certain that building the image for another CPU architecture will not cause different behaviour?