Built on a vision that the latest software is the most secure, software updates are central to the FoundriesFactory security story. What happens, however, when software delivery is compromised? Is recovery complicated? Enter TUF key rotation!
TLDR
Securing the Internet of Things
Loss of trust and diminished reputation have become the norm for companies who fail to protect their customers sufficiently from data leaks and intrusion. This on top of fines levied for undesired and unexpected security incidents. But why "unexpected"? The list of those affected by large. and. expensive. data. breaches. include. mature. software. companies. that, for a long time coming, have known of the perils of lax security. Surely, the measures they'd established were formidable. Surely, they expected constant, even increasing, attempts to circumvent said measures as their business grew. Surely.
Security is just hard, and it's getting harder as infiltrators learn new ways to duck and confuse. It is for this reason that Foundries.io has put emphasis on keeping software up to date. It's true that the latest software will have the latest security measures, and while making updates possible to each iteration of IoT and edge device has been the impetus of the FoundriesFactory, keeping IoT software up to date is the raison d'être.
Keeping software up to date is not a novel idea. The majority have already accepted the shift to automatic software updates, because it has been shown time and again that the worst hack is one where the user is at fault for neglecting to apply existing security patches. Hackers have adapted to this environment of automatic software updates by attacking the software delivery mechanism. Successful intrusion means preventing users from ever receiving software updates, or worse, it could mean reverting deployed security patches. That's why keeping software up to date means preventing hackers from tampering with the software update mechanism so that software stays secure as intended. It's no surprise then that companies such as Docker, CoreOS, Datadog, Python, and Foundries.io (to name a few) include The Update Framework (TUF) as one of the measures to ensure that their customers receive what they expect.
How does TUF prevent software delivery from being compromised though? What steps need to be taken to ensure that it works as intended for a FoundriesFactory?
How to Secure Your Factory for OTA Updates With TUF Keys
When a FoundriesFactory is created, novice users may be tempted to begin registering newly flashed devices. In fact, there is nothing stopping one from making modifications to the Linux microPlatform (LmP) source code repository. The CI service that was bootstrapped during FoundriesFactory creation will dutifully take those changes, publish the generated artifacts, and the OTA Connect service will make signed updates available to registered devices as soon as the OTA Connect client checks in. This is a very good setup and displays the ease that the LmP affords users who require that only trusted updates be installed. And TUF helps make this possible!
┌─ changes trigger jobs ─┐
│ │
│ ▼
┌─────────────────────┐ ┌─────────────────────┐
│ │ │ │
│ SOURCE CODE │ │ CONTINUOUS │
│ REPOSITORY │ │ INTEGRATION │
│ │ │ │
└─────────────────────┘ └─────────────────────┘
│
┌─ generated artifacts ─┘
│ .─.
▼ .─. ( ).─.
┌─────────────────────┐◀─────────── ( .─.`─'( .─.
│ │ ───────────▶─ .─.
│ OTA UPDATE │◀───────── ( Registered ( )
│ SERVICE │ ───────────▶. Devices .─.'
│ │◀────────── ( )
└─────────────────────┘ ────────────▶─' .─. ( `─'
`─( )─'`─'
`─'
Fig. 1: LMP artifacts deployed to registered devices
This process can be improved, however, if the FoundriesFactory owner rotates the root offline keys. TUF root key rotation is encouraged as the next step after creating a FoundriesFactory because it gives owners confidence that they alone control the root offline keys that secure their devices' updates.
NOTE: rotation is also needed in the event that the root of trust is compromised
How to Rotate TUF Keys
Download TUF root keys archive
In order to perform a TUF root key rotation, the original root keys are required. They may be downloaded from the URL provided in the notification email for each newly created FoundriesFactory.
Be aware, once keys are downloaded, Foundries.io will permanently delete the resource from its servers
Install Garage Deploy tool
To do key rotation, use Here.com's Garage Sign tool (it's bundled with Garage Deploy). Garage Sign manages the root of trust, and owners may use it to rotate root keys. For convenience (and sanity), this article will use the containerized version available from the Foundries container registry. If Docker is already installed, test that garage-sign works like so:
docker run -it hub.foundries.io/aktualizr garage-sign --help
Usage: garage-sign [user-keys|delegations|init|key|move-offline|root|targets|export-credentials|get-targets|verify-root] [options]
--help prints this usage text
--version
-h, --home-dir <value>
--verbose
--version
Command: user-keys [gen|id] [options]
manage keys not associated with a specific repository
...
Feel free to instead follow the Garage Sign install instructions provided by Here.com.
With this output present, both Docker and the aktualizr image are prepared.
Rotate TUF root keys with Garage Sign
Now that Garage Sign is installed and the TUF root keys archive is available, rotation can begin. With Garage Sign, this is an 8 step process:
- log into the Docker container with a mounted volume to save changes locally
mkdir rotation_artifacts mv /path/to/tuf_archive /path/to/rotation_artifacts docker run -it --rm -v /path/to/rotation_artifacts:/rotation_artifacts hub.foundries.io/aktualizr sh cd rotation_artifacts
all subsequent steps are assumed to happen within the same Docker session from this step
- Pull the current
targets.jsonfrom the OTA Connect service using thetufrepodirectory from the TUF root keys archive:garage-sign root pull --repo /path/inside/tuf/archive/tufrepo - Generate new root keys
garage-sign key generate --repo /path/inside/tuf/archive/tufrepo --type rsa --name <new-key-name>
- Add the new key to the TUF manifest
garage-sign root key add --repo /path/inside/tuf/archive/tufrepo --key-name <new-key-name>
- Remove the previous key from the TUF manifest
garage-sign root key remove --repo /path/inside/tuf/archive/tufrepo --key-id <previous-key-id> --key-name <previous-key-name>
- Sign the changes with both the new and previous keys
garage-sign root sign --repo /path/inside/tuf/archive/tufrepo --key-name <new-key-name> --key-name <previous-key-name>
- Push the signed changes to the OTA Connect service
garage-sign root push --repo /path/inside/tuf/archive/tufrepo
- Store the updated offline TUF root keys securely
Do a little dance! The TUF root keys have now been rotated successfully!
Rotate TUF root keys with fioctl
How about no dance at all? A misstep in the above could have dire effects, for instance, if step 8 was neglected. But wait!
There is a better way for a FoundriesFactory, Raymond: the fioctl tool! Fioctl uses the same process, but in one step:
fioctl keys rotate /path/to/tuf/archive
With this one command, TUF root keys have been rotated and the local TUF root key archive has been overwritten with the updated manifest.
Maintaining Your TUF Key Rotation For OTA Updates
Some time later, after learning how to rotate keys, FoundriesFactory owners will move on and continue working on the next big thing. Probably, some Friday afternoon rolls around. Everything is following the "happy path": LmP devices, via the OTA Connect Client, are routinely checking in to the OTA Connect service. Each communication between these entities inspects the TUF manifest for changes and validity before trusting any accompanying notification. Everything passes inspection and, today, intended devices are notified of a new build's availability. As the new version is installed without a hitch, the weekend draws nearer.
┏━OTA UPDATE SERVICE━━━━━━━━━━━━━━┓◀──checkin request─── ┏━LMP Registered Device : Foo━━━━┓
┃ ┃ ┃ ┃
┃ available device builds ┃ ┃ aktualizr-Lite ┃
┃ - Foo: 764 (new!) ┃ ┃ - validate TUF manifest √ ┃
┃ - Bar: 800 ┃ ┃ - check for platform update √ ┃
┃ ┌─────────────────┐ ┃ ┃ ┌───────────────┐ ┃
┃ │TUF credentials │ ┃ ┃ │TUF credentials│ ┃
┃ └─────────────────┘ ┃ ┃ └───────────────┘ ┃
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛───TUF response─────▶ ┗━━━━━━━━━━━━━━━━━━━━━━━Build 763┛
Fig. 2: Device, "Foo", is notified of Build 764's availability
Some time in the night, when everyone's singing karaoke (or whatever the weekend brings) and no one is watching, a bad actor attempts a man-in-the-middle attack. Suddenly, the ostensible OTA Connect service notifies the Foo devices of another update (one that would revert the day's fixes).
┏━Fake OTA UPDATE SERVICE━━━━━━━━━┓◀──checkin request─── ┏━LMP Registered Device : Foo━━━━┓
┃ ┃ ┃ ┃
┃ available device builds ┃ ┃ aktualizr-Lite ┃
┃ - Foo: 765 (new!) ┃ ┃ - validate TUF manifest x ┃
┃ ┃ ┃ - check for platform update x ┃
┃ ┃ ┃ ┌───────────────┐ ┃
┃ ┃ ┃ │TUF credentials│ ┃
┃ ┃ ┃ └───────────────┘ ┃
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛──Bad TUF response──▶ ┗━━━━━━━━━━━━━━━━━━━━━━━Build 764┛
Fig. 2: Device, "Foo", is notified of Build 765's availability by an impersonator
These hypothetical Foo devices would soon find that communication fails TUF validation and reject the update. Neither the devices, nor the customers would be affected, and karaoke singers the world round could continue their rendition of "Get Schwifty".
Hackers are a persistent bunch, though. What if they somehow compromised the TUF credentials? What now?! As a Cromulon might have once said, show them what you've got! Rotate those TUF credentials, the IoT world is counting on it.