FoundriesFactory TUF Keys Rotation

Built on a vision that the latest software is the most secure, software updates are central to the FoundriesFactory security story. What happens, however, when software delivery is compromised? Is recovery complicated? Enter TUF key rotation!


fioctl is magic

Hard and Scary

Loss of trust and diminished reputation have become the norm for companies who fail to protect their customers sufficiently from data leaks and intrusion. This on top of fines levied for undesired and unexpected security incidents. But why "unexpected"? The list of those affected by large. and. expensive. data. breaches. include. mature. software. companies. that, for a long time coming, have known of the perils of lax security. Surely, the measures they'd established were formidable. Surely, they expected constant, even increasing, attempts to circumvent said measures as their business grew. Surely.

Security is just hard, and it's getting harder as infiltrators learn new ways to duck and confuse. It is for this reason that has put emphasis on keeping software up to date. It's true that the latest software will have the latest security measures, and while making updates possible to each iteration of IoT and edge device has been the impetus of the FoundriesFactory, keeping IoT software up to date is the raison d'être.

Keeping software up to date is not a novel idea. The majority have already accepted the shift to automatic software updates, because it has been shown time and again that the worst hack is one where the user is at fault for neglecting to apply existing security patches. Hackers have adapted to this environment of automatic software updates by attacking the software delivery mechanism. Successful intrusion means preventing users from ever receiving software updates, or worse, it could mean reverting deployed security patches. That's why keeping software up to date means preventing hackers from tampering with the software update mechanism so that software stays secure as intended. It's no surprise then that companies such as Docker, CoreOS, Datadog, Python, and (to name a few) include The Update Framework (TUF) as one of the measures to ensure that their customers receive what they expect.

How does TUF prevent software delivery from being compromised though? What steps need to be taken to ensure that it works as intended for a FoundriesFactory?

In the beginning

When a FoundriesFactory is created, novice users may be tempted to begin registering newly flashed devices. In fact, there is nothing stopping one from making modifications to the Linux microPlatform (LmP) source code repository. The CI service that was bootstrapped during FoundriesFactory creation will dutifully take those changes, publish the generated artifacts, and the OTA Connect service will make signed updates available to registered devices as soon as the OTA Connect client checks in. This is a very good setup and displays the ease that the LmP affords users who require that only trusted updates be installed. And TUF helps make this possible!

           ┌─ changes trigger jobs ─┐
           │                        │
           │                        ▼
┌─────────────────────┐  ┌─────────────────────┐
│                     │  │                     │
│     SOURCE CODE     │  │     CONTINUOUS      │
│     REPOSITORY      │  │     INTEGRATION     │
│                     │  │                     │
└─────────────────────┘  └─────────────────────┘
           ┌─ generated artifacts  ─┘
           │                              .─.
           ▼                         .─. (   ).─.
┌─────────────────────┐◀─────────── (  .─.`─'( .─.
│                     │ ───────────▶─             .─.
│     OTA UPDATE      │◀───────── (   Registered (   )
│       SERVICE       │ ───────────▶.  Devices   .─.'
│                     │◀────────── (                )
└─────────────────────┘ ────────────▶─' .─.  (   `─'
                                     `─(   )─'`─'

Fig. 1: LMP artifacts deployed to registered devices

This process can be improved, however, if the FoundriesFactory owner rotates the root offline keys. TUF root key rotation is encouraged as the next step after creating a FoundriesFactory because it gives owners confidence that they alone control the root offline keys that secure their devices' updates.

NOTE: rotation is also needed in the event that the root of trust is compromised

How to rotate keys

Download TUF root keys archive

In order to perform a TUF root key rotation, the original root keys are required. They may be downloaded from the URL provided in the notification email for each newly created FoundriesFactory.

Be aware, once keys are downloaded, will permanently delete the resource from its servers

Install Garage Deploy tool

To do key rotation, use's Garage Sign tool (it's bundled with Garage Deploy). Garage Sign manages the root of trust, and owners may use it to rotate root keys. For convenience (and sanity), this article will use the containerized version available from the Foundries container registry. If Docker is already installed, test that garage-sign works like so:

docker run -it garage-sign --help
Usage: garage-sign [user-keys|delegations|init|key|move-offline|root|targets|export-credentials|get-targets|verify-root] [options]
  --help                   prints this usage text
  -h, --home-dir <value>
Command: user-keys [gen|id] [options]
manage keys not associated with a specific repository

Feel free to instead follow the Garage Sign install instructions provided by

With this output present, both Docker and the aktualizr image are prepared.

Rotate TUF root keys with Garage Sign

Now that Garage Sign is installed and the TUF root keys archive is available, rotation can begin. With Garage Sign, this is an 8 step process:

  1. log into the Docker container with a mounted volume to save changes locally
    mkdir rotation_artifacts
    mv /path/to/tuf_archive /path/to/rotation_artifacts
    docker run -it --rm -v /path/to/rotation_artifacts:/rotation_artifacts sh
    cd rotation_artifacts

all subsequent steps are assumed to happen within the same Docker session from this step

  1. Pull the current targets.json from the OTA Connect service using the tufrepo directory from the TUF root keys archive:
    garage-sign root pull --repo /path/inside/tuf/archive/tufrepo
  2. Generate new root keys
garage-sign key generate --repo /path/inside/tuf/archive/tufrepo --type rsa --name <new-key-name>
  1. Add the new key to the TUF manifest
garage-sign root key add --repo /path/inside/tuf/archive/tufrepo --key-name <new-key-name>
  1. Remove the previous key from the TUF manifest
garage-sign root key remove --repo /path/inside/tuf/archive/tufrepo --key-id  <previous-key-id> --key-name <previous-key-name>
  1. Sign the changes with both the new and previous keys
garage-sign root sign --repo /path/inside/tuf/archive/tufrepo --key-name <new-key-name> --key-name <previous-key-name>
  1. Push the signed changes to the OTA Connect service
garage-sign root push --repo /path/inside/tuf/archive/tufrepo
  1. Store the updated offline TUF root keys securely

Do a little dance! The TUF root keys have now been rotated successfully!

Rotate TUF root keys with fioctl

How about no dance at all? A misstep in the above could have dire effects, for instance, if step 8 was neglected. But wait!

<slams fist on table>
"There must be a better way!"
- Raymond Hettinger

There is a better way for a FoundriesFactory, Raymond: the fioctl tool! Fioctl uses the same process, but in one step:

fioctl keys rotate /path/to/tuf/archive

With this one command, TUF root keys have been rotated and the local TUF root key archive has been overwritten with the updated manifest.


Some time later, after learning how to rotate keys, FoundriesFactory owners will move on and continue working on the next big thing. Probably, some Friday afternoon rolls around. Everything is following the "happy path": LmP devices, via the OTA Connect Client, are routinely checking in to the OTA Connect service. Each communication between these entities inspects the TUF manifest for changes and validity before trusting any accompanying notification. Everything passes inspection and, today, intended devices are notified of a new build's availability. As the new version is installed without a hitch, the weekend draws nearer.

┏━OTA UPDATE SERVICE━━━━━━━━━━━━━━┓◀──checkin request─── ┏━LMP Registered Device : Foo━━━━┓
┃                                 ┃                      ┃                                ┃
┃ available device builds         ┃                      ┃ aktualizr-Lite                 ┃
┃  - Foo: 764 (new!)              ┃                      ┃  - validate TUF manifest √     ┃
┃  - Bar: 800                     ┃                      ┃  - check for platform update √ ┃
┃ ┌─────────────────┐             ┃                      ┃ ┌───────────────┐              ┃
┃ │TUF credentials  │             ┃                      ┃ │TUF credentials│              ┃
┃ └─────────────────┘             ┃                      ┃ └───────────────┘              ┃
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛───TUF response─────▶ ┗━━━━━━━━━━━━━━━━━━━━━━━Build 763┛

Fig. 2: Device, "Foo", is notified of Build 764's availability

Some time in the night, when everyone's singing karaoke (or whatever the weekend brings) and no one is watching, a bad actor attempts a man-in-the-middle attack. Suddenly, the ostensible OTA Connect service notifies the Foo devices of another update (one that would revert the day's fixes).

┏━Fake OTA UPDATE SERVICE━━━━━━━━━┓◀──checkin request─── ┏━LMP Registered Device : Foo━━━━┓
┃                                 ┃                      ┃                                ┃
┃ available device builds         ┃                      ┃ aktualizr-Lite                 ┃
┃  - Foo: 765 (new!)              ┃                      ┃  - validate TUF manifest x     ┃
┃                                 ┃                      ┃  - check for platform update x ┃
┃                                 ┃                      ┃ ┌───────────────┐              ┃
┃                                 ┃                      ┃ │TUF credentials│              ┃
┃                                 ┃                      ┃ └───────────────┘              ┃
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛──Bad TUF response──▶ ┗━━━━━━━━━━━━━━━━━━━━━━━Build 764┛

Fig. 2: Device, "Foo", is notified of Build 765's availability by an impersonator

These hypothetical Foo devices would soon find that communication fails TUF validation and reject the update. Neither the devices, nor the customers would be affected, and karaoke singers the world round could continue their rendition of "Get Schwifty".

Hackers are a persistent bunch, though. What if they somehow compromised the TUF credentials? What now?! As a Cromulon might have once said, show them what you've got! Rotate those TUF credentials, the IoT world is counting on it.

Keep up to date with