The world of self-updating binaries is a bit of a dumpster fire. I recently took a dive into the mess hoping to give our users something they can trust.
Since its inception, there has been a never ending debate taking place about how to distribute updates of Fioctl®. There's no universally accepted way to do things like this. Your rich uncle might employ a small army of people to try and:
- Package it for Windows under one of their many package managers
- Package it for macOS with Homebrew
- Package it for Debian, Ubuntu, RedHat, Fedora, Arch, Nix, etc, etc, etc.
Another interesting approach has grown out of some of the more modern "cloud native" command line interfaces, which also inspired Fioctl. These tools tend to be written in Golang in such a way that makes static linking for a given OS and CPU easy. These projects publish those static binaries with their GitHub releases. This is how Fioctl started…and it's not the worst thing. However, the problem comes when you want users to update the tool.
This year, I started looking at teaching Fioctl how to update itself. The key algorithm is determining if there's an update. Ironically, as I started on how I wanted this work, I rediscovered why TUF really matters. The naive approach starts with: "I'll look at the latest GitHub release. If that's newer, I should get it". Then I started to think, "I should include an 'MD5SUMS' file so I can verify what I downloaded is correct". Then I started to think, "I need to GPG sign this file to keep things safe". Then I started thinking about GPG keys and wondering how a once promising life had led me to this point. But then I realized — "Andy — you've spent the past 7 years of your life solving this very problem for other people with TUF".
TUF was invented to keep people like me from coming up with bad ideas.
As I looked around at other similar projects, I noticed that nobody was doing this. I also realized that most of this could be managed in GitHub without the need for TUF "repo servers". This idea is generically useful to many projects, so I thought it deserved a blog.
Utilizing TUF + GitHub + go-tuf + Golang embed
Fioctl as a project works like it always has. However, we now have a special branch, tuf-metadata. In there is a repository directory that can be accessed through GitHub's raw file API like:
https://raw.githubusercontent.com/foundriesio/fioctl/tuf-metadata/repository/targets.json
I'd done some hacking on the go-tuf project before, and found it to have a great community. I decided to use this implementation for the Fioctl logic to find updates.
The real trick with TUF metadata is keeping the online signatures up-to-date. For this, we use a simple GitHub action that runs periodically. Now after every release of Fioctl, one of the maintainers will add the release to our targets.json.
Another nice trick we employed was the management of the initial root metadata. A common way to deal with this is with "trust on first use" where you more or less blindly trust the first copy of root.json you download. Newer versions of Golang include a neat feature, embed, that allows you to embed arbitrary content into a Golang binary. We use this feature to embed our root.json into Fioctl.
Teaching Fioctl to Self-Updating Golang Binary
With TUF metadata in place, Fioctl just needed to learn how to use this data. We wanted to start very conservatively and just have fioctl version display if there was an update available. If so, it also shares the command to run to update itself. The updater wound up being pretty simple.
I wrote a "FioctlUpdateFinder". It compares the version of Fioctl running to the latest version found in targets.json. If there's a new version, it returns a "FioctlUpdate" object that has single method "Do()".
This method downloads the binary, validates its hash, and replaces itself. Of course, nothing is ever easy on Windows, so I had to get a little creative there. I'm not proud of that hack, but am in good company as many projects use this technique.
In the end, we've got a secure way for Fioctl to update itself. As far as security goes, it's fairly straightforward to understand.
Nobody else is doing anything quite like this that I know of, but I'd be happy for people to steal bits of this code and start doing something similar.