Starting with Linux microPlatform update 0.13, we are including support for over-the-air (OTA) updates.
After extensive research, discussion in the embedded Linux community and a review of the current Linux OTA solutions, we concluded that update systems should be protected by comprehensive update frameworks that are designed to protect against known exploits at a minimum.
Through advanced key management practices, TUF (the update framework) and UPTANE (TUF principles adapted to safety critical / automotive embedded systems) provide solid foundations against many of today's most malicious attacks.
- Arbitrary installation attacks.
- Endless data attacks.
- Extraneous dependencies attacks.
- Fast-forward attacks.
- Indefinite freeze attacks.
- Malicious mirrors preventing updates.
- Mix-and-match attacks.
- Rollback attacks.
- Slow retrieval attacks.
- Vulnerability to key compromises.
- Wrong software installation.
During our investigations we started working with the team at Advanced Telematics (acquired by HERE technologies in 2018) and started learning about their update platform that was used in Automotive Grade Linux, "ATS Garage". In January 2018, the team released their platform as an open source project, OTA Community Edition and we went to work integrating with the system as it provides, what we believe is the highest level of security for embedded system software updates.
We won't go into all of the details right now, but you can find out more from the following links:
- Documentation available at: https://docs.atsgarage.com/usage/devices.html
- Source code available at: https://github.com/advancedtelematic/ota-community-edition
- As this support is part of a subscriber update, you'll need to setup a subscription account on https://app.foundries.io
- Get a supported hardware development platform (Raspberry Pi 3, DragonBoard-410c, DragonBoard-820c, Toradex Colibri iMX7D, Solidrun HummingBoard 2, Compulab IoT Gate, Beaglebone Black Wireless, x86)
- Download the latest system image: https://app.foundries.io/mp/lmp/latest/artifacts/ (0.13 and later will be OTA ready)
- Install the Linux microPlatform
- Upload an OS to ATS Garage for deployment
- Provision your device
- Upload a second OS to ATS Garage
- Deploy the new OS to your device(s)
- Reboot to activate the new image
To get started updating your Linux microPlatform devices follow these instructions:
Install the Linux microPlatform
NOTE: You must use update 0.13 or newer builds with the OSTree repo
Follow the documentation to install the Linux microPlatform. With a running Linux microPlatform system, follow the next steps to provision your device on ATS Garage.
Create an ATS Garage Account
Create an ATS Garage account at https://app.atsgarage.com/ .
Create your ATS Garage credentials
Generate your auto provisioning credentials at https://app.atsgarage.com/#/profile/access-keys
Once you generate your credentials you can use the same credentials on many devices. These credentials are used to generate device-specific keys during initial device provisioning and to sign the OSTree repository when you upload the repo to ATS garage.
Push the Linux microPlatform image to ATS garage
You can use a command line tool to sign and publish the artifacts of a build to ATS Garage (e.g. for Raspberry Pi 3)
Create a clean directory with your credentials.zip file and the OSTree Repo
$ mkdir lmp $ cd lmp
Download your ATS Garage credentials file (https://app.atsgarage.com/#/profile/access-keys) and make it available in a local folder
Download and extract microPlatform OSTree repository tarball from a specific LMP build number (e.g. https://app.foundries.io/mp/lmp/0.13/artifacts/supported-raspberrypi3-64/other/raspberrypi3-64-ostree_repo.tar.bz2)
$ tar -jxvf raspberrypi3-64-ostree_repo.tar.bz2
Identify the image version ID from the running LMP system if provisioning your device for the first time
- This is required to avoid a version mismatch between the running system and the uploaded image
- In a future update this step will not be required
raspberrypi3-64:~$ sed -e "s/.*lmp-//" -e "s/\"}//" /var/sota/installed_versions 283
Set the version ID for the uploaded image
- Same as described in the previous step if provisioning for the first time
- When uploading additional image updates just use the microPlatform update version instead (e.g. 0.14)
Run our image publishing script using a docker container in order to sign and push the image to ATS Garage
$ docker run --rm -it -v $PWD:/build --workdir=/build opensourcefoundries/aktualizr \ ota-publish -m raspberrypi3-64 -c credentials.zip -r ostree_repo -v $VERSION
In case of upload errors (e.g. OSTree fetch error: 502), please make sure to retry a few more times as the server might be overloaded.
Verify the Package can be viewed on the ATS Garage
Browse to: https://app.atsgarage.com/#/packages/raspberrypi3-64-lmp
Now that we have an update loaded into ATS garage, we will provision the device, we are using the Raspberry Pi 3 for demonstration purposes, though the other boards can also be used.
Copy and start the Aktualizr client daemon
on your host machine:
$ scp credentials.zip [email protected]:~/
on your target machine: (ssh to the system)
$ sudo mv credentials.zip /var/sota/sota_provisioning_credentials.zip $ sudo cp /usr/lib/sota/sota_autoprov.toml /var/sota/sota.toml # Note: Aktualizr will start automatically once it finds /var/sota/sota.toml
Browse to your ATS Garage account and manage the device's root filysystem
If everything goes right, your device will auto-register to the ATS Garage system. This may take a few minutes.
You can begin managing the system the moment you upload the next update. At Foundries.io we produce OS updates about every 10 days, so one will be available soon.
Once you upload a second image to ATS Garage, you can initiate an OS upgrade from the ATS web system and then reboot the target device to activate the image.
Further documentation will be available soon.
Issues and observations
Watching the aktualizr logs
If you are having a problem with the device connecting to ATS garage, you can watch the client logs with the following commands
sudo journalctl -f -u aktualizr.service
Device fails to show up on ATS garage
We have recently noticed a variety of timeouts when communicating with ATS garage. From our experience, most of these timeouts and communication failures will self-recover. If you notice a delay in device communication or registration you can review the aktualizr client logs (see above).
A sample set of errors that seem to be recoverable may look like:
Apr 26 17:15:18 raspberrypi3-64 aktualizr: curl error 28 (http code 0): Timeout was reached Apr 26 17:15:19 raspberrypi3-64 aktualizr: curl error 35 (http code 0): SSL connect error
Device fails to sync and kRejectAll observed
We have noticed that from time to time, the aktualizr client delivered in update 0.13 is susceptible to an issue that has been fixed in 0.14 and later microPlatform updates.
The issue, which has been fixed in the upstream aktualizr client, puts the device into a state where it can not take an update.
To work around this issue you can use the following commands to modify the SQLite database.
# stop the aktualizr service sudo systemctl stop aktualizr.service # remove the 'meta' table from the sqlite database docker run -v /var/sota/sql.db:/sql.db opensourcefoundries/utilities sqlite3 /sql.db "drop table meta;" # start the aktualizer service sudo systemctl start aktualizr.service # OPTIONAL: you can watch the aktualizr logs to verify the connection sudo journalctl -f -u aktualizr.service