Securing your Production Updates with FoundriesFactory

In the previous blog post we talked about releasing on demand with FoundriesFactory Waves. Now, let's talk about how you can take a complete ownership of what is inside your Factory releases by protecting them with your cryptographic signatures.

Background

When you create a Wave, we store a special kind of TUF targets role on our servers, which encapsulates your ability to Release on Demand just a single version of Factory's software components. This prevents new software versions built by a CI/CD pipeline from automatically reaching your production devices.

Each TUF role created by either a CI pipeline or a Wave is signed by a corresponding TUF role's key to protect it from being altered. However, if all the keys used to sign these roles were owned by Foundries.io, you still couldn't feel fully safe. Naturally, you want to be an owner of those keys.

Thus, with the FoundriesFactory Waves feature we give you the ability to sign your Factory's production TUF targets role with your own digital signature. This ability complements previously delivered FoundriesFactory features which allow you to own your device gateway PKI and TUF root keys. Together, these features give you a full control of what and when is delivered to your production devices according to the best industry practices.

Securing your Updates with your Digital Signature

Using Andy's words from his blog post on taking ownership of device gateway PKI: "security requires precision". Often, that precision requires an exact order of exact actions.

Below are those exact steps needed to secure updates delivered from your Factory to your production devices. These steps must be taken in an exact order as they appear below to give you the best protection. Thus, Foundries.io tools and web services make sure to guide you through these actions in a proper order.

Taking Ownership of your Factory's TUF Root Role Keys

Your first action after you've created your Factory is usually to take ownership of your Factory's TUF root key. For that, you need to download your TUF keys archive and rotate root keys with fioctl keys rotate-root <factory-keys.tgz> command as described in TUF root keys blog post. Once you are done with this step, you can be sure that nobody can take ownership of your Factory.

Note that a fioctl keys rotate-root went through a major refactoring compared to the original blog post. In particular, we needed to support a production use case which required us to remove a dependency on the garage-sign tool. This also means that an old way to rotate your Factory's TUF root role keys with the help of garage-sign is no longer supported.

Taking Ownership of your Factory's TUF Targets Role Keys

We at Foundries.io believe that before you decide to make your first release using Waves, you need to take ownership of your Factory's TUF targets role keys. This warrants that nobody except you are technically capable of altering your production TUF targets. This guarantees that you have the exclusive control of the software updates that are delivered to your production devices on behalf of your Factory.

Thus, after playing with your Factory for a while but before going to production, you need to run a fioctl keys rotate-targets <factory-keys.tgz>. This command generates a new TUF targets role key for your Factory and saves it into your local <factory-keys.tgz> file. It also generates and uploads a new TUF root role pointing to this new TUF targets role key. Naturally, this command also signs that new TUF root role with your TUF root role key.

Releasing on Demand with Waves Protected by your Digital Signature

Now (after your Factory's TUF roles keys are setup) you can plan your release strategy. Any time you want to release a new version of your Factory's software component(s) to your production devices you would create a new Wave and sign an associated TUF targets role with your TUF targets role key. This is normally done by running a fioctl wave init <wave-name> <version> [<tag>] -k <factory-keys.tgz command.

In the previous blog post on using Waves to Release on Demand we intentionally omitted this -k <factory-keys.tgz> parameter to make things easier to digest, as in that blog post we focused on the mechanics of how a Wave works. However, this is a mandatory argument and you wouldn't be able to create a new Wave without signing its targets with your TUF targets role key. That way you are warranted that there is no way to deliver updates to your production devices without your digital consent in a form of your TUF targets role key signature.

TL;DR: Some Neat Technical Details

Additional Measures to Protect your Updates agains Attack Vectors

There is a small yet important difference in how we handle TUF root versus targets role signatures at Foundries.io. For TUF targets role we wanted to introduce an additional protection, because securing updates to your devices is (in our opinion) the most important part of TUF. For that we require any TUF targets role delivered to your production devices to be signed by both your (offline) TUF targets role key and an additional server-side (online) TUF targets role key. We apply that online signature when you create a Wave. Needless to say, an online TUF targets role key is unique to your Factory, we don't share it across several factories.

All in all this way gives you a huge security benefit: it makes much harder for a malicious user to take control of your Factory's software updates. For example, in order to alter your Factory's TUF targets role a malicious user would need to do one of the following:

  • take possession of both your (offline) TUF targets role key and Foundries.io server-side (online) TUF targets role key associated with your Factory.
  • take possession of both your (offline) TUF targets role key and your admin user credentials associated with your Factory.
  • take possession of both your TUF root role key and your admin user credentials associated with your Factory.
  • take possession of your TUF root role key and take ownership of your devices networking infrastructure.

Given that we believe that all of your Factory credentials must be kept in utter secrecy as do our server-side access credentials, it must be extremely hard to break through either of the above attack vectors.

What about Targets Produced by CI/CD

At Foundries.io we always try to find a good balance between security versus convenience. Thus, we don't require an offline TUF targets role key signature to be present in your CI Targets (those produced by your Factory's CI pipelines). In contrast, it should only be signed by an online TUF targets role key during a CI pipeline. This gives you the convenience of instantly delivering updates to your development devices straight from a CI build just like before. And these updates are still reasonably secured by our Foundries.io PKI individually configured for your Factory.

This flexibility (secure production updates and convenient CI updates) is possible by a unique Foundries.io approach to your Factory TUF roles: we persist a separate set of all TUF roles for your CI versus production needs, each enforcing different security policies.

Re-signing your Production TUF Targets Role on Key Rotation

Obviously, you will need to rotate your Factory's TUF targets role key from time to time according to your security policy, it is not meant to be a one-time procedure. When you do this, all current production targets need to be re-signed using a new key. Thus, a fioctl keys rotate-targets command will do this routine task for you in an secure atomic way with zero downtime for your Factory's updates.

The only limitation is that we don't allow to rotate your Factory's TUF targets role key during an active Wave. But you would usually not notice this, because a Wave rollout is meant to be a short-lived procedure.

Keep TUF Root versus Targets Role Keys Separately

Although by default we generate both TUF Root and Targets role keys in one file, nothing stops you from storing them separately if your security policy requires that. This is easily accomplished by the following workflow:

  • make a backup of your <factory-keys.tgz> file after you have generated a new TUF targets role key (by a fioctl rotate-targets <factory-keys.tgz> command).
  • untar your <factory-keys.tgz> file into a safe temporary location.
  • move TUF targets keys into a separate temporary location (preserving the same directory structure).
  • create two separate tar archives:
    • one from an original location (will only contain your TUF root role keys);
    • another one from a separate location where you moved your TUF targets role keys (this one will apparently only contain your TUF targets role keys).
  • if everything went well and you are happy with the result, remove files from a temporary location as well as the backup file.

Now, you can name those two files e.g. root-keys.tgz and targets-keys.tgz. The first one will be rarely used (only when you need to rotate your Factory TUF roles' keys), but it must be stored in an extremely safe location, because it's a root chain of trust of your factory. By "extremely safe" we mean not just "protected", but also "reliable", because losing access to this file literally means losing ownership of your Factory. Thus, make sure to keep as many copies of those file as you need to warrant its durability.

The second file (targets-keys.tgz) will be used much more often, literally every time you need to release a new version of your Factory's software components to production devices. Thus, it should be more accessible, although it still needs a reasonable amount of access protection. Depending onto your security policy you might also provide this file to a dedicated set of your personnel which are authorized to release new software updates. We also recommend providing a reasonable amount of durability to this file (like backups), but losing access to it isn't the end of your Factory. As long as you keep access to your Factory's TUF root key you can generate a new TUF targets key with the help of a fioctl key rotate-targets <root-keys.tgz> command (using the same workflow described above).

A cool thing about separating root vs targets keys is that a fioctl wave init command only really needs your TUF targets role keys. Thus, an operator should be able to start a new Wave by using just targets-keys.tgz from the above example. For instance, this command should work as expected: fioctl wave init new-wave 42 -k <targets-keys.tgz> command. This way your TUF root role key becomes less exposed and as a result more protected.

Keep up to date with Foundries.io