OTA updates in an embedded Linux environment: a developer’s guide

Photo of Raul Muñoz

Posted on Dec 19, 2024 by Raul Muñoz

7 min read

Gather ‘round, developers, and listen to a chilling tale of digital terror. In the shadowy realm of the internet the Mirai botnet lurks ‐ a nightmare born from the most mundane of vulnerabilities. Picture innocent devices: security cameras, routers, digital sentinels waiting to be corrupted. In October 2016, Mirai unleashed a devastating assault that brought internet giants trembling to their knees.

The monsters weren’t complex hackers, but the simplest form of digital demons: forgotten updates and unchanged credentials. This is why over-the-air (OTA) updates are more than a convenience ‐ they’re your shield against the lurking terrors of the connected world.

But there is yet another terror which lies beyond in the deepest dark. Picture this: your connected device attempts to install your latest lovingly crafted security update to prevent horrors like Mirai ‐ but something goes awry. It is left isolated, cold, alone.

“It’s bricked,” your coworker mutters with a resigned sigh. As a shiver goes up your spine, the status lights turn red and the phone starts ringing.

Every Linux® embedded developer has horror stories like this. It’s part of the reason OTA updates are sometimes seen as contentious and even more frighteningly ‐ optional. It’s viewed as complex and difficult to implement in embedded devices, which typically have comparatively small amounts of compute, power, and network bandwidth available.

But, in fact, there’s no reason to back off from implementing OTA updating – open-source tools and services for running security-conscious OTA updates efficiently are ready to hand and they are built into the FoundriesFactory platform.

Let’s dive in and understand why OTA updating is important, how OTA updates are created and delivered, and which are the best resources to use when implementing OTA updating in your next project.

Reliable delivery of patches and enhancements

The clue is in the name: OTA updates allow software to be updated wirelessly and remotely; that means there’s no need for a technician to attend on-site, the device does not have to be recalled to a service lab, and the user does not have to physically connect their device with a wire to download an update.

This is convenient and saves on operating expenses. But the benefit of OTA updating goes further than that: with OTA updating in place, you – the OEM – control how updates are delivered and installed. You’re not reliant on the user to run the updating process. So that means you can keep a fleet of devices updated and help to protect them from threats, reducing the risk that a rogue or negligent user leaves their device vulnerable. And that vulnerability might not only affect that one user’s device – it could be leaving a gate open for threats to reach the entire fleet.

This type of lateral movement attack is particularly dangerous because it allows attacks to remain undetected while they expand their control and access sensitive data. OTA updating is a sure way to mitigate the risk of such an attack and keep an entire fleet of devices protected.

Take care of the security of the updates themselves

OTA updates are critical for device security, but the update process itself can become a significant vulnerability if not carefully designed. An insecure update mechanism could inadvertently become a gateway for malware distribution across entire device fleets. Attackers have developed numerous sophisticated techniques to exploit update systems. Imagine a scenario where an attacker can prevent your device from recognizing an available update, or worse, trick your system into installing an older, vulnerable version of software while making it appear new and secure.

Cryptographic verification is a cornerstone of a security-focused update strategy. By implementing comprehensive metadata tracking, version validation, and sophisticated authentication mechanisms, we can transform updates from a potential security weakness into a method of improving device protection. The key is to create a system that can definitively prove an update's origin, verify its integrity, and ensure it hasn't been tampered with during transmission.

This approach goes beyond simple file transfer. It's about creating a multi-layered security strategy that anticipates and neutralizes potential threats before they can compromise a single device ‐ let alone an entire fleet of connected systems.

Best practices in OTA updating

We’re fortunate that we don’t have to figure all this out ourselves. In fact, folks have been safely delivering OTA updates for years now, and we know what works and what doesn’t. These are some of the most important best practices in OTA updating:

  • Take care of the security and integrity of OTA updates: use encrypted connections and digital signatures to protect the authenticity and integrity of updates. Make sure that your embedded devices implement a secure boot mechanism, and that they verify the integrity of the update file before installing it, to prevent a cyber-attacker from injecting unauthorized modifications.
  • Optimize the update process to give minimal disruption to the user experience: while an update is being downloaded and installed, the device might be out of action, which can be inconvenient for the user. So make the update process as efficient and reliable as it can be. One thing you can do is to implement ‘delta updates’ — that is, updates which only install changes to a software stack, rather than replacing the stack entirely. This reduces the size of the update file and takes up less network bandwidth. You should also have fail-safe mechanisms, such as rollback options and dual partitions, so the user can revert to a known good state if problems occur during or after an update.
  • Monitor OTA update deployments: you’ll need to make sure that the update process runs as planned. Collect and analyze relevant data to identify potential problems, and track the progress of the update. Log problems and failures, have a reporting mechanism, and allow for remote debugging to make sure you can troubleshoot updated devices when needed.

How security-focused OTA updating is supported in the FoundriesFactory platform

Foundries.​io offers a comprehensive OTA update framework for security-focused OTA updating of embedded devices. The OTA features provided in the FoundriesFactory platform include The Update Framework (TUF) service for a security-rich update delivery. We use Mutual TLS (mTLS) and provide tools for the device to verify certificates ‐ you can read more about our implementation and the impact this has on overall OTA security on our documentation site.

The FoundriesFactory platform enables updates not only for applications but for the entire operating system and firmware. Combining the Linux microPlatform operating system with the FoundriesFactory platform provides a comprehensive solution that allows you to perform atomic, immutable, security-focused and incremental updates over-the-air (as well as offline if required).

Because the OTA update capability is built into the platform you are using to manage your entire product development, your system’s OTA update features can be developed from the beginning of your project. That makes it easier for you to test your update mechanism, and make sure that it is compatible with the hardware and software configuration of your product.

On top of that, you’ll be using the FoundriesFactory CI/CD process and software bill-of-materials (SBOM) generation tooling to help identify which licenses, software versions, and patches are applied to your devices, helping to mitigate supply chain attacks and CVE exposure.

If you would like to learn more about how Foundries Factory can help improve the security of your OTA updates then book a call with a member of the team today!

Related posts