OTA updates are a defining feature and core benefit of using FoundriesFactory. New software rolls out to devices in the field as soon as CI builds complete. What if devices that receive updates at less-than-ideal times?
A device could be in the middle of some important, non-interruptible task when the update notification shows up. Installing an update at a random time can cause a break in the application, or worse, damage the device itself. A way to prevent that from happening come by using aktualizr-lite callbacks. A callback is any executable that gets called when update related events happen on the device. The events reflect the phases of the OTA update: downloading, installing, rebooting. It’s probably most convenient to use a shell script as a callback. Aktualizr-lite provides more details about the event by means of environment variables: MESSAGE and RESULT. MESSAGE provides information about current phase in OTA process and RESULT gives additional information about status of this or previous phase.
As described in the testing blog post, LAVA is used for automated testing of LmP. Since LAVA dispatcher runs as a containerized application on the devices in the LAB it can use the callback mechanism to prevent stopping running jobs. This would happen during OTA update (OS or containers) as aktualizr-lite is not aware of any internal states of container applications.
Setting up a callback can be done from within a container application. There are a few steps to make it happen:
Provide access to important directories
Container application needs to write files to
/var/sota/and/etc/sota/conf.d/directories. The latter is where aktualizr-lite takes it’s configuration snippets from. The former is a convenient place to host the callback script. This can be done as a part of docker-compose.yaml
volumes:
- /etc/sota/conf.d/:/etc/sota/conf.d/ # for installing .toml
- /var/sota/:/var/sota/ # for installing callback
Copy configuration snippet and the callback script during container startup
Restart aktualizr-lite so the changes are picked up
An example configuration snippet can be composed as follows (lava-callback.toml):
[pacman]
callback_program = "/var/sota/check-running-jobs.sh"
It’s important to know that aktualizr-lite only supports one callback script as of the publishing of this post.
An example callback script can be composed as follows:
#!/bin/bash
if [ "${MESSAGE}" = "install-pre" ]; then
echo "Checking if worked is running LAVA jobs"
# check if lava is running any jobs
while docker exec lava-dispatcher-lava-dispatcher-1 ps -ef | grep lava-run; do
echo "Waiting for the running jobs to complete"
sleep 5
done
fi
This is the script for the lava-dispatcher application running in Foundries.io testing LAB.
The example above prevents interruptions in LAVA test jobs caused by OTA updates. When complicated logic is required, the next step is to create a custom SOTA client.