Integrating with AWS IoT using Just-in-Time Provisioning

Integrating with AWS IoT Core sounds easy at first. However, making it work for a fleet of devices turns out to be a complicated task of navigating many sources of information... this week I found clarity.

AWS IoT Core is a pretty neat offering that gives fleet operators inroads into all things AWS. On the plus side - they have thought this out and offer the things you need for doing it the right way. However, nothing about "the right way" is very clear or easy to try out. To be fair, I am an embedded person now working in the cloud and our cloud provider is not AWS. I may just have the wrong mindset.

After hours of reading, grasping at straws in videos, and decoding JSON serialized into strings embedded into other JSON that is then escaped for Bourne shell, I started to understand what I needed. Then I stumbled upon a pretty well written guide.

In the end, I found this type of work does not translate well to copy/paste CLI commands and instead created a Python script that can be (ab)used.

What You Need to Know

If you are working with a fleet, you need a secure way to provision devices. AWS IoT has a smart approach based on mutual TLS (mTLS). They also let you own the Certificate Authorities they'll trust.

There are two ways to register new devices "at scale". The first one is called Just-in-Time Registration (JITR). JITR is cool because you can do anything. It is not cool because there is extra work (a Lambda function and wiring things together) and cost (you pay for the time spent running Lambdas).

The newer approach is called Just-in-Time Provisioning (JITP). This one is easier to configure, cheaper to operate, and is the method I have chosen.

How It Works

There are 3 pieces to the puzzle. The first piece is the paperwork of defining all the rules and policies for IAM. Most material I found on this has you clicking buttons on the web UI. I did this in the script to save you the trouble and mistakes.

The second part is getting AWS to trust your CA(s). They have a simple process where you prove you are the owner of a CA by signing a registration code they provide.

A FoundriesFactory includes an easy way to get PKI set up for your fleet. The "local-ca" used there will be used in the manufacturing process. Thus, you have the CA and all your production devices will have a client certificate (/var/sota/client.pem) that was signed by this CA. Once AWS knows about this CA, it can allow factory devices to establish mTLS connections.

The third part is the JITP logic which informs AWS how to create a Thing upon the initial device connection. In our case, we tell AWS to create a Thing named from the device's UUID (Common Name attribute of the device certificate). The Thing is allowed to perform any iot actions.

NOTE - A device's first connection will fail. In the background JITP kicks in. Subsequent connections will succeed.

How to Do It

I have created a simple Python3 script to configure AWS from your host system containing your Factory's device-gateway PKI files. The script requires vanilla python (no special libraries) and the AWS CLI installed and configured.

You will need to update the factory_cas variable at the top of the file. In most cases, you will only have a single key-pair and not two like me. If you know a thing or two about AWS, you might want to make some tweaks to lines 5-22 as well. Then you simply run the script once.

At this point, you will need a device that's been provisioned with this CA. We have a Factory Registration Reference Server that works with lmp-device-register for doing this. As a result of the device registration, a device key and certificate are created (pkey.pem and chained.pem referenced further/below). You also need to copy the AWS CA to /var/sota/aws-ca.pem. You can then run MQTT from a container on the device to publish a message:

$ docker run --rm -it -v /var/sota:/var/sota:ro -w /var/sota eclipse-mosquitto \
	mosquitto_pub \
		-h <YOUR endpoint printed out by the script> \
		--cafile ./aws-ca.pem \
		--cert ./client.chained \
		--key ./pkey.pem  \
		-p 8883 \
		-t testtopic/$(hostname) \
		-d \
		-m "{\"unix-ts\": $(date +%s)}"

Remember — the first time you run this it will fail. However, aws iot list-things will show your device. You can run the command again and a message will publish to MQTT. The AWS IoT web interface includes a "Test" link in the left-hand menu that allows you to subscribe to MQTT topics and watch messages:

Wrapping Up

There's a lot more that could be done with this. I have had ideas like:

• Configuring collectd to publish data over MQTT to AWS.
  • Have it enabled/disabled via fioctl.
  • Have an MQTT subscriber put that data in CloudWatch metrics
• Experiment with Device Shadows
• Doing some DataDog-ish monitoring without paying per device fees.

Stay tuned, I am hoping to write about doing this same thing but using a commercially available board that has a hardware security module on it.