Fleet Management

How to wave

In a production environment it is convenient to plan updates, and have control on when and which device(s) are updated. FoundriesFactory Waves is the feature for this.

This post gathers a suggested sequence of commands to configure the first wave, and to guide for next waves.

The main purpose of a Wave is to plan the fleet update, giving the Factory operator full control on when to allow certain devices to look for a new OTA update available. It also allows the operator to split up the fleet of devices according to the defined update plan.

The wave feature is first presented in the post Release on demand with FoundriesFactory Waves which details the description of a wave and how to configure different approaches to deployment of waves.

The wave feature depends on configuration of device groups, a feature presented in the post FoundriesFactory Device Groups

NOTE: Requirement

  • device group feature is added on fiotcl Release v0.14
  • wave feature is added on fioctl Release v0.15
  • production feature is added on lmp-device-register Release mp-80

Create a FoundriesFactory

The instructions to create a FoundriesFactory are detailed in the Getting Started page.

When a new FoundriesFactory is created, an email with a link to download the offline keys are sent to the registered email.

Download the offline keys and store them in a secure way. It is a tarball also known as root credentials. They are used for TUF key management.

Configure the local environment

For this example, the following variables are being exported:

host:~$ export PATH_TO_KEY=/media/secure/offline-creds.tgz
host:~$ export FIOCTL_FACTORY=my_factory_name

Where PATH_TO_KEY is the absolute path to the offline credential tarball file, and FIOCTL_FACTORY is the FoundriesFactory name.

Check the access to the factory

Login fioctl to the Foundries.io server with the following command. This page detail how to login.

host:~$ fioctl login

Rotate the the offline keys

host:~$ fioctl keys rotate-root $PATH_TO_KEY
host:~$ fioctl keys rotate-targets $PATH_TO_KEY

Detailed information for key rotation can be found here.

Work with the FoundriesFactory

Work with the source code until the targets available are ready to be published to the devices on a production environment. Flash the devices with the desired target.

For help on how to flash the device the details are provided in this document.

Register the production device

Make sure the device is registered as PRODUCTION=ON:

device:~$ sudo su
device:~$ export PRODUCTION=ON
device:~$ lmp-device-register -n <device-name>

Where <device-name> is unique for every device registered.

Register as many devices as needed.

Populate the initial Target

host:~$ fioctl wave init -k $PATH_TO_KEY populate-targets <version> <tag>
host:~$ fioctl wave complete populate-targets

Where <version> is the target version, a number that can be found on the target page for the factory: https://app.foundries.io/factories/factory-name/targets/

And <tag> is the production tag that is used for the wave.

Details on the production targets can be found at the OTA Reference Manual.

The targets access the FoundriesFactory through the Device Gateway which is presented in Taking ownership of device gateway PKI.

Configure the fleet following the update plan

In this example, there are 3 devices, and they are split into 3 different device groups. However, in a production scenario it is common to cluster the devices by architecture, region, or Internet access.

The main point of clustering the devices in a device group is to coordinate the update on all those devices at once.

Create the device groups

For example, create three device groups in an easy to follow order with this example.

host:~$ fioctl config device-group create one "the first set"
host:~$ fioctl config device-group create two "the second set"
host:~$ fioctl config device-group create three "the third set"

The OTA Reference Manual also details the device configuration.

Assign the devices with the device groups

In this example there are three devices: deviceA, deviceB and deviceC. Each device is part of a device group. Each device group has only one device.

For a real production environment, a device group can comprise several devices, however a device can be configured on only one device group.

host:~$ fioctl device config group deviceA one
host:~$ fioctl device config group deviceB two
host:~$ fioctl device config group deviceC three

Create an update

After some time, changes are produced with new features and/or bug fixes. A new target is then created, and the update to the devices can be planned.

It is time for a wave!

Create a wave

At this point, the environment is ready to run a wave to update the registered devices in an organized way.

The command line to create wave1 is:

host:~$ fioctl wave init wave1 <version_> <tag> -k $PATH_TO_KEY

The wave1 is created, but no device has yet been updated. See the list of waves so far with the following command:

host:~$ fioctl wave list
NAME               VERSION    TAG         STATUS    CREATED AT                  FINISHED AT
----               -------    ---         ------    ----------                  -----------
wave1              <version_> <tag>       active    2021-04-06T14:40:48.855063
populate-targets   <version>  <tag>       complete  2021-04-05T18:33:37.477752  2021-04-05T18:34:06.946635

See the status of the current active wave with the following command

host:~$ fioctl  wave status
Wave 'wave1' for tag <tag> version <version_> is active

Created At: 	2021-04-06T14:40:48.855063
Device Groups on Tag '<tag>':  3
Device Groups Rollout:         0
Devices on Tag '<tag>':        3
Devices Updated:               0
Devices Scheduled for Update:  0
Devices Not Scheduled:         3

GROUP  TOTAL  UPDATED  NEED UPDATE  ONLINE  ROLLOUT AT
-----  -----  -------  -----------  ------  ----------
one    1      0        1            1       At Wave Completion
three  1      0        1            1       At Wave Completion
two    1      0        1            1       At Wave Completion

Rollout an update for a device group

The next step is to configure the devices of a certain device group to look for an update.

In this example, the first device group to receive a rollout is one.

host:~$ fioctl wave rollout wave1 one

Continue the wave

From this point, the wave continues according to update plan defined previously by the factory operator.

For this example, the sequence is rollout device group two, and then device group three, using the following commands:

host:~$ fioctl wave rollout wave1 two
host:~$ fioctl wave rollout wave1 three

However, another option would be rollout all the remaining devices instead:

host:~$ fioctl wave complete wave1

Or maybe, cancel this wave and have no more devices updated with <version_> is also an option.

host:~$ fioctl wave cancel wave1

The last three possible flows for the wave are only a illustration of how flexible the update path can be. There are other options and configurations described in the OTA Reference Manual.

helpful status commands:

host:~$ fioctl -f <factory> wave list
host:~$ fioctl -f <factory> wave status
host:~$ fioctl -f <factory> devices list
host:~$ fioctl -f <factory> targets list

Reference Links

Related posts

Keep up to date with Foundries.io