IoT device management, applied to embedded devices based on a Linux® operating system (OS), is an important function for maintaining the performance and security of a fleet of devices in the field: it is the set of functions which allows for over-the-air (OTA) software upgrades and security patches throughout the life of an IoT or edge product.
IoT device management is an umbrella term: it encompasses a range of functions which require capabilities built into devices during the embedded software development phase, and which are performed as part of the manufacturing of the device, and after delivery to the end customer, when the device is in the field.
The functions which enable IoT device management are:
IoT device security Helps protect IoT devices and data transfers from cyber-attacks. Key elements of embedded systems security include; secure boot, to ensure that the device is running the correct software, use of strong cryptographic algorithms for key generation, revocation and rotation, secure storage for sensitive data or credentials, secure data communications, and secure OTA updates helps ensure that a known and authorized source can only update the device.
Device provisioning In the production of an embedded Linux device, provisioning is the process by which the manufacturer installs a unique and tamper-proof root of trust into each device. This root of trust underpins the operation of the public key infrastructure (PKI) used for end-to-end security.
Secure onboarding and offboarding Onboarding ensures that devices are secured and authenticated to an original equipment manufacturer (OEM) or service provider when first powered up, without having to set up each device individually on the production line. For large deployments, zero-touch onboarding saves time and eliminates the risk of third-party interference, which can create an exposure to cyber-attacks.
Device configuration This might be required after installation, to enable new services, to add features, or to restrict certain users’ or devices’ access to features.
Applications and services management As part of the device lifecycle management process, the manufacturer or service provider might need to add, remove or change applications or services that are running on the device.
Software updates OTA updates are the main means by which the manufacturer or service provider maintains the security of an IoT or edge device in the field. These may be required to fix bugs, to provide security patches in response to a critical vulnerability or exposure (CVE) notice, or to introduce new functionality or features.
In a large fleet, IoT device management also benefits from enabling subsets of the fleet to be used as test devices or canaries to ensure that update packages work as expected in the field. This means that the fleet is likely to be updated over a period of time, avoiding the risk of flooding a network by updating millions of devices simultaneously.
Remote access This might be necessary for product testing, analysis, or servicing. Any remote access system should be secure by design to ensure additional threat vectors are not introduced.
Change of ownership When a device changes hands, it might require a remote factory reset, to ensure personal information is removed.
End of life or decommissioning The final stage of device lifecycle management is when the device has reached the end of its life, when it should be safely and securely decommissioned. An important part of the decommissioning process is to remove the device from the manufacturer’s or service provider’s fleet configuration. Secure decommissioning also removes personal information from the device.
Typically, every one of these components depends on secure remote access to individual devices and/or entire fleets; for large installations this has to be an automated process. In a DevOps for IoT system such as the FoundriesFactory™ platform, IoT device management systems are built in and fully integrated into the development flow.
This means that embedded device OEMs can undertake an industrial IoT device development project confident that, once the device goes into production and devices are shipped to end users, updates and patches can be easily and securely delivered and installed to the entire fleet.
Just like Apple performs device management for global fleets of devices running the iOS operating system, or Google for smartphones based on the Android™ operating environment, manufacturers of automotive, industrial, commercial or consumer embedded devices can enable a comprehensive IoT device management service which can be applied globally to a fleet of devices based on a Linux OS.
The challenges in managing embedded and IoT devices
In a small fleet of 10, 20 or even 100 devices, management and maintenance are relatively simple and straightforward. It is likely that all devices will share an identical or very similar Linux OS configuration, and delivery and installation of updates can be managed and monitored individually by the IoT device management team.
But at scale, IoT device management becomes much more complex and difficult to perform. There are various reasons for this.
Demand for bandwidth and strain on the network: Distributing software updates to thousands, or even millions, of devices simultaneously can overwhelm network infrastructure. This can lead to network congestion, potential network failures, and delayed deployments.
When pushing a large firmware update to a geographically dispersed fleet, different devices will enjoy differing levels of network connectivity. IoT device management has to allow for the likelihood that a global update will in practice, be delivered and installed at different times in different parts of the world. In addition, some countries or areas might suffer network failures that delay deployment of a security-critical update, creating a temporary geographical disparity between one part of the fleet and another.
Device diversity and compatibility: A large fleet will often contain a diverse set of hardware and software in devices from different generations of the same basic device design, or in devices with different feature sets tailored to different regional requirements or customer segments.
Diversity often extends to the Linux OS on which embedded devices run. Linux OS implementations are continually refined and improved to take advantage of new features, to gain better performance or to secure the device against new threats. Large fleets of devices can therefore commonly include devices running many different versions of the Linux OS, as well as different versions of firmware and application software.
Ensuring compatibility across this diverse ecosystem adds a layer of complexity for the IoT device management system to handle. The need to test and validate OTA updates on each device type before deployment can be a substantial drain on development resources and time.
Update scheduling and rollback: The co-ordination of updates across a large fleet requires careful scheduling to minimize disruption to operations.
The manufacturer or service provider also needs to enable a rollback mechanism to allow for reversion to a previous version in the event of an update failure. This can complicate IoT device management, because it leads to a situation in which some parts of a fleet are running later versions of the system software than others.
When operating across multiple time zones, it is complicated to schedule updates at times of low use.
Security vulnerabilities: The more devices in a fleet, the larger the attack surface available to cyber attackers. The problem is compounded because a failure to update a single device could compromise the entire fleet.
Compromising the reliability of OTA updates Across a large fleet of IoT devices, network interruptions, unexpected power outages, or even device malfunctions during the update process can cause device bricking or data corruption. The OTA update process requires the successful delivery of the update package to each device, installation, and reporting back that installation has taken place. The larger the fleet, the more likely it is that update failures will occur, requiring intervention by the IoT device management system.
Device lifecycle management: Managing the entire lifecycle of devices, from provisioning and deployment to decommissioning, calls for advanced systems in large fleets to track device inventory, manage configurations, and ensure compliance with regulations.
Embedded device manufacturers are under strong pressure to provide effective IoT device management, not least because of the need to comply with laws and regulations which touch on security. Among the most widely applied of these is the European Union’s Cybersecurity Resilience Act: this requires the manufacturer – not the owner or a service provider – to take a wide-ranging set of measures to build security protection into an embedded device, and to enable maintenance of security protection for the lifetime of the device. Among these measures are the ability to deliver and implement software updates (either over-the-air or via a wired connection), and the ability to maintain an updated software bill-of-materials (SBOM). The SBOM enables the manufacturer to determine which units are vulnerable to any emerging threat listed in a new CVE notice.
For the automotive industry specifically, a widely applied standard is ISO/SAE21434:2021, which specifies engineering requirements for cybersecurity risk management in the electrical and electronic systems in road vehicles, including their components and interfaces.
IoT device management provides essential underpinning for both the software update capability and the SBOM capability required by cybersecurity regulations and standards.
Critical components of a secure IoT device management strategy
A good IoT device management system answers three basic questions about a fleet of IoT devices:
- Does every production unit have its own unique identifier, and is this securely protected and only known to authorized users?
- Can every production unit securely receive and install the software updates required to maintain its protection against known and emerging cybersecurity threats?
- Does the manufacturer have continually updated information about the status of every production unit, including its SBOM, its security update status, and its performance?
IoT device management for a fleet of embedded Linux OS-based devices requires a complex set of processes, software and services to address each of these three questions.
1. Device provisioning and identity management
For a manufacturer to successfully monitor and update a device, it needs to know the identity of that device. Cyber attackers seek to frustrate and undermine cybersecurity protection by tampering with the identity of devices or by interfering with authentication processes. If they are successful in this, a cyber attacker could for instance, substitute a spoof device for a genuine device, and thus gain access to secure credentials such as network authorization.
Secure bootstrapping is the process by which a manufacturer establishes and protects a device’s unique identity. The elements of a secure bootstrapping process include:
Establishing a root of trust: Secure bootstrapping creates a root of trust in the device's hardware. This is a foundational element that cannot be easily tampered with because it is based on a highly secure hardware component such as a hardware security module (HSM) or trusted platform module (TPM).
This root of trust is used to verify the integrity of subsequent software components loaded at start-up, a process known as secure boot.
Preventing unauthorized software execution: By verifying digital signatures and checksums, secure bootstrapping helps ensure that only authorized firmware and operating systems are allowed to run. This prevents malicious actors from loading compromised or altered software onto the device.
Protecting against firmware attacks: Firmware is a prime target for attackers, as it controls the device's core functionality. Secure bootstrapping helps mitigate the risk of firmware tampering, which could lead to device hijacking or data breaches.
Certificate-based device authentication: Secure bootstrapping should incorporate a means for verifying a device's identity during the secure boot process. This is crucial for preventing device spoofing and ensuring that only legitimate devices can connect to networks and services.
Certificate-based authentication is a highly secure method of authentication: it relies on digital certificates, a public key, and a digital signature to verify a device’s identity. To be authenticated, the device presents a digital certificate to the manufacturer’s server. The server verifies the certificate's validity against a trusted list of certificates or a Certificate Authority (CA). If the certificate is valid, the device is authenticated.
Preventing supply chain attacks: Secure bootstrapping can help mitigate risks associated with supply chain attacks, in which malicious software is injected into devices during production or distribution to the end user. By verifying software integrity at boot time, it can detect and prevent the execution of such malicious software.
To enable these bootstrapping processes, the manufacturer needs to program security resources, such as keys and a unique ID, into secure memory in each production unit. This is the process of provisioning, and the process itself needs to be carefully protected, as a cyber attacker who gains access to secrets such as keys can compromise an entire fleet of devices.
Cybersecurity professionals keenly support zero-touch provisioning because it:
- Reduces the scope for human error in provisioning
- Automates the application of consistent security policies
- Ensures uniformly secure configuration of devices in production
- Allows for tight central control of the provisioning process
- Minimizes the need for manual intervention in provisioning, thus reducing the provisioning process’s exposure to attack
- Supports proven and systematic compliance with cybersecurity regulations such as the Cybersecurity Resilience Act
- Minimizes exposure to supply chain attacks
2. Delivery and installation of OTA updates
OTA update systems provide a more secure means of updating a device remotely. With an effective OTA update system in place, device manufacturers can respond quickly to new vulnerabilities and exposures, protecting a fleet of devices before a cyber attacker has the chance to exploit a new known vulnerability.
The most important elements of a secure OTA update system are:
- Secure boot based on a hardware root of trust, and a chain of trust extending from the hardware root of trust to the bootloader, OS, and application firmware, ensuring that each component is verified before execution.
- Cryptographic verification, based on digital signatures and backed by security audits and penetration testing. Audits should be conducted regularly to identify and remedy potential vulnerabilities in the update server infrastructure.
- Device authentication and authorization.
3. Monitoring and analysis of a fleet of devices
Monitoring and logging of a fleet ensures that the device manufacturer can track the update status of all devices, identify devices which have suffered from failed update events, and take corrective action. This means that the manufacturer should track the status of updates on each device, and keep detailed logs of all update activities, including successful and failed updates, to facilitate troubleshooting and security analysis.
Logs also support anomaly detection: anomalies can often be the result of a malicious attack on a device. Early detection of an anomaly in a device’s operation allows the manufacturer to rapidly deploy counter-measures to protect it, halting wider damage to the fleet before the attack can be extended.
Analysis of devices also allows for optimal resource utilization. Embedded devices, unlike traditional computers, often have limited processing bandwidth, memory, and power. These constraints can make it difficult to implement robust security systems without compromising performance or functionality.
To mitigate the difficulties, security solutions for embedded systems should be lightweight and efficient, using techniques such as:
- Lightweight cryptography
- Efficient protocols
- Resource-aware algorithms
Effective device lifecycle management requires that, throughout the life of a device, from design and deployment through to decommissioning, the manufacturer should implement consistent processes and proven tooling for the authentication, updating and monitoring functions, which are essential to lifetime security.
Accelerating development and deployment with a DevOps approach
The globally competitive industry for embedded devices requires manufacturers to design and release products within shorter and shorter development windows, to get to market with new features and improved performance before rivals saturate the market.
Effective IoT device management can contribute to the acceleration of product development and deployment. For example, it can enable manufacturers to go to market early with a product design that has a limited feature set, and extend its features over time, in response to customer feedback, by delivering OTA updates which contain feature enhancements.
A DevOps for IoT approach to software development, in which the development and operational arms of a manufacturer are tightly integrated, to allow for the rapid deployment to devices in the field of newly developed software supports this accelerated release schedule.
When security software updates are treated as a core element of the product development process, this approach might also be known as DevSecOps.
Core elements of the DevOps/DevSecOps approach are: CI/CD for embedded systems.
CI/CD automates build, testing, and deployment processes, substantially reducing the time it takes to release new features and updates. CI/CD also enables more frequent and smaller releases, allowing developers to incorporate feedback quickly and adapt to changing requirements. This helps accelerate time to market.
A CI/CD process also contributes to the improvement of software quality through:
- Automated testing – unit testing, integration testing, and system testing ensure that code changes are thoroughly validated, reducing the risk of bugs and errors.
- Early bug detection – allows developers to address issues before they escalate and become more costly to fix.
- Consistent testing – ensures that all code changes are subjected to the same rigorous standards.
CI/CD also provides for enhanced collaboration across the development, production and maintenance teams, through the integration of version control systems, and by making the development process transparent to all team members.
Lastly, CI/CD results in more reliable and less risky product designs and deployments. Automated deployment reduces the risk of human error, ensuring that software updates are deployed consistently and reliably, while rollback capabilities in CI/CD pipelines allow developers to quickly revert to a previous working version of the software in case of errors.
CI/CD pipelines can also be configured to include hardware-in-the-loop testing. This allows for automated testing of the software in a simulated or real hardware environment. CI/CD minimizes development risk because it promotes the release of smaller, incremental code changes, which are easier to test and manage, reducing the risk of major failures.
Manufacturers that adopt a DevSecOps approach can further reduce risk, because here, security features and cybersecurity risk analyses can be integrated into the development process from the start of a product development. This means that essential security capabilities, such as secure boot, OTA updating and an SBOM, are implemented as early as possible, ensuring that the hardware resources that they need are built into the product’s architecture.
By contrast, it is much harder to provide the proper hardware resources and development time needed for cybersecurity protection if they are not considered until near the end of the development cycle, when decisions might already have been made that militate against the effective implementation of security protection functions.
How the FoundriesFactory platform supports IoT device management needs
Cybersecurity measures are high on the list of priorities for Embedded Linux OS-based device manufacturers, a response both to laws such as the EU’s Cybersecurity Resilience Act, and to the need of manufacturers to protect the valuable intellectual property (IP) embedded in their devices.
IoT device management is a foundational element of cybersecurity, as it enables crucial security features such as OTA updating, device monitoring, security-focused provisioning and device authentication to be scaled over large fleets of geographically dispersed devices.
The key elements of an IoT device management system have been described in this article. To implement effective IoT device management, manufacturers can use the FoundriesFactory DevOps for IoT platform, which has been specially made for Linux OS-based embedded devices. It is a security-rich and integrated set of development, testing, production, deployment, maintenance and decommissioning tools. By basing their development, deployment and maintenance operations on the FoundriesFactory IoT development platform, OEMs benefit from:
- Faster time to market
- Reduced development and operating costs
- Enhanced device and data security
Supporting CI/CD processes, the platform enables rapid prototyping and production-scale management of devices.
Key IoT device management facilities built into the FoundriesFactory platform include:
- Security-focused OTA updating – devices automatically check in with the FoundriesFactory servers to check for the availability of new ‘targets’ (update packages). Update delivery benefits from the security features embedded in The Update Framework (TUF), the utility which the FoundriesFactory platform uses.
The update service can readily be configured and customized using the outward-facing APIs in the FoundriesFactory platform. A simple command line interface (CLI) enables technicians to operate devices remotely in the field.
The platform also provides for efficient update and device management processes through its provision of a ‘group’ facility. This enables the manufacturer to define groups of devices with shared characteristics. It can improve the efficiency of updating and other processes by selecting a target or update, for delivery to a group rather than to an entire fleet.
The Waves functionality in the FoundriesFactory updating service enables the manufacturer to deploy an update to a select group of devices, and then monitor the deployment to check for anomalies or problems in the field. When the update’s operation has been verified, Waves lets the manufacturer deploy the same update to the rest of the group, or to an entire fleet, without re-delivering the update to the original devices used for the test deployment.
CI/CD processes to streamline building, testing, and deploying both embedded Linux OS distributions and containers for applications and services, speeding up the development cycle.
Consistent security practices including the provision of an SBOM for every production unit. The SBOM is continually updated to take account of all software updates, and provides a source and version information for every software package integrated into the device.
IoT device management can be improved throughout the embedded device lifecycle when OEMs adopt the FoundriesFactory platform.