Google Assistant on the LmP

In this blog, I describe how to run the Google Assistant SDK on a Raspberry Pi with an external speaker and microphone. After you get things configured and gather key pieces of information you'll have a simple, portable and easy to reproduce Google Assistant appliance.

Running the Google SDK on the LmP is extremely simple, just run the following docker command and you'll have the demo running in a matter of minutes.

docker run --rm -it --device /dev/snd
-e MIC_ADDR=$MIC_ADDR -e SPEAKER_ADDR=$SPEAKER_ADDR -e PROJECT_ID=$PROJECT_ID
-e MODEL_ID=$MODEL_ID -e CLIENT_SECRET=$CLIENT_SECRET -e CLIENT_ID=$CLIENT_ID
-e REFRESH_TOKEN=$REFRESH_TOKEN
--name ok-google-cli-demo opensourcefoundries/ok-google

Ok, maybe the challenge isn't running the sample, but figuring out the hardware addresses and Google Cloud variables necessary.

In the rest of this blog, we'll show you where to collect all of these values so that running the Google Assistant will be easy, repeatable and portable.

Hardware needed

Raspberry Pi 3b (others LmP devices will work, but for this demo we'll use the 3b)
1. All the peripherals (Ethernet cable, 5V micro-usb power, 8GB microSD card)
External speaker and Microphone
1. $85 - JABRA 510 USB/Bluetooth speaker phone (speaker & mic)
Other Hardware tested
1. $12 - USHonk USB Speaker (speaker only)
2. Headphones (speaker only)
3. $5 - Kinobo USB mic (mic only)

Load the latest Linux microPlatform

Get all the latest software by using the latest microPlatform release. Sign up for a subscription @ https://app.foundries.io and download the latest artifacts.

Download a LmP pre-built binary for your Raspberry Pi 3b, write it to a microSD card or decompress and write it to the flash device using the dd command.

If you are unfamiliar with the Linux microPlatform, more detailed instructions on using the Linux microPlatform can be found at Documentation on how to install the Linux microPlatform

1. Connect to your Raspberry Pi using SSH

ssh [email protected]

2. Find your microphone and speaker addresses

In order to route the audio to the right device we will need to find the hardware addresses (and possibly adjust the volume a bit).

Plug your device(s) into your Raspberry Pi (we will be using the Jabra Device)
Load and run a container that contains the ALSA subsystem (not this also happens to be the container we'll use to run the Google Assistant, but for now we are going to use it only to find the HW device IDs)
1. Find the microphone device (Note: Card: 2 and Device: 0) (unfortunately, the card ID may may change over reboots)

docker run -it --rm --device /dev/snd opensourcefoundries/ok-google arecord -l
setup authorized credentials file
setup alsa configuration
+ exec arecord -l
**** List of CAPTURE Hardware Devices ****
card 2: USB [Jabra SPEAK 510 USB], device 0: USB Audio [USB Audio]
Subdevices: 1/1
Subdevice #0: subdevice #0

Find the speaker device (Note: Card: 2 and Device: 0) (unfortunately, the card ID may may change over reboots)

docker run -it --rm --device /dev/snd opensourcefoundries/ok-google aplay -l
setup authorized credentials file
setup alsa configuration
+ exec aplay -l
**** List of PLAYBACK Hardware Devices ****
card 0: vc4hdmi [vc4-hdmi], device 0: MAI PCM vc4-hdmi-hifi-0 []
  Subdevices: 1/1
  Subdevice #0: subdevice #0
card 1: ALSA [bcm2835 ALSA], device 0: bcm2835 ALSA [bcm2835 ALSA]
  Subdevices: 7/7
  Subdevice #0: subdevice #0
  Subdevice #1: subdevice #1
  Subdevice #2: subdevice #2
  Subdevice #3: subdevice #3
  Subdevice #4: subdevice #4
  Subdevice #5: subdevice #5
  Subdevice #6: subdevice #6
card 1: ALSA [bcm2835 ALSA], device 1: bcm2835 ALSA [bcm2835 IEC958/HDMI]
  Subdevices: 1/1
  Subdevice #0: subdevice #0
card 2: USB [Jabra SPEAK 510 USB], device 0: USB Audio [USB Audio]
  Subdevices: 1/1
  ÅSubdevice #0: subdevice #0

Now we know that the MIC is 2,0 and the Speaker is 2,0 we will be able to pass this information into the container for further adjustments. (Note: the card ID may may change over reboots)

Test the speaker volume
1. Note: we pass in MIC_ADDR and a SPEAKER_ADDR environment variables
2. Note: we also call the 'speaker-test' utility and run the wav test. You should hear "Front Left"

docker run -it --rm --device /dev/snd -e MIC_ADDR="hw:2,0" -e SPEAKER_ADDR="hw:2,0" opensourcefoundries/ok-google speaker-test -t wav

Press CTRL-C to exit
Adjust the speaker volume, you can also adjust the MIC gain using the alsamixer application

docker run -it --rm --device /dev/snd -e MIC_ADDR="hw:2,0" -e SPEAKER_ADDR="hw:2,0" opensourcefoundries/ok-google alsamixer
  OR
docker run -it --rm --device /dev/snd -e MIC_ADDR="hw:2,0" -e SPEAKER_ADDR="hw:2,0" opensourcefoundries/ok-google alsamixer --card=2 --view=all

After you adjust the volume, you can retest using speaker-test command from step 1 above.

3. Authorize your device with the Google Cloud services

Note: Much of this blog was excerpted from Google's SDK documentation at https://developers.google.com/assistant/sdk/guides/library/python/

Configure an Actions Console project

A Google Cloud Platform project, managed by the Actions Console, gives your device access to the Google Assistant API. The project tracks quota usage and gives you valuable metrics for the requests made from your device.

To enable access to the Google Assistant API, do the following:

Open the Actions Console.
Click on Add/import project.
To create a new project, type a name in the Project name box and click CREATE PROJECT. If you already have an existing Google Cloud Platform project, you can select that project and import it instead.
Click the Device registration box.
Enable the Google Assistant API on the project you selected (see the Terms of Service). You need to do this in the Cloud Platform Console.
ENABLE THE API in Console.developers.google.com
Then Click Enable.

Set activity controls for your account

In order to use the Google Assistant, you must share certain activity data with Google. The Google Assistant needs this data to function properly; this is not specific to the SDK.

Open the Activity Controls page for the Google account that you want to use with the Assistant. You can use any Google account, it does not need to be your developer account.

Ensure the following toggle switches are enabled (blue):

Web & App Activity
- In addition, be sure to select the Include Chrome browsing history and activity from websites and apps that use Google services checkbox.
Device Information
Voice & Audio Activity

Register the Device Model

Use the registration UI in the Actions Console to register a device model.

Open the Actions Console.
Select the project you created previously.
Select the Device registration tab (under ADVANCED OPTIONS) from the left navbar.
Click the REGISTER MODEL button.

Create model

Fill out all of the fields for your device. See the device model JSON reference for more information on these fields.
When you are finished, click REGISTER MODEL

Download credentials file

The credentials.json file must be located on the device. Later, you will run an authorization tool and reference this file in order to authorize the Google Assistant SDK sample to make Google Assistant queries. Do not rename this file.

Download this file and transfer it to the device.

scp ~/Downloads/credentials.json [email protected]:/home/osf/

4. Run the SDK and Sample code

In the Google samples you would now install software onto your target device, however the container you have been using to query the audio hardware is configured and ready to run the sample with no further modification.

Authorize your device

In order to connect your device to the Google services you will first need to generate the appropriate credentials. We'll do this in a more interactive version using the bash shell and mapping in our credentials.json file from above.

docker run -it --rm --device /dev/snd -v ${PWD}/credentials.json:/credentials.json opensourcefoundries/ok-google bash google-oauthlib-tool --scope https://www.googleapis.com/auth/assistant-sdk-prototype
--scope https://www.googleapis.com/auth/gcm
--save --headless --client-secrets /credentials.json

Running google-oauthlib-tool will return a HTTPS URL that you will need to copy and paste into your browser to complete the authorization steps.

Copy the authorization code back into your terminal window to complete the authorization dance and get your credentials.

Note three values from your credentials file

cat /root/.config/google-oauthlib-tool/credentials.json

{ "client_secret": "kF4ZLu-X", "scopes": [ "https://www.googleapis.com/auth/assistant-sdk-prototype", "https://www.googleapis.com/auth/gcm" ], "token_uri": "https://accounts.google.com/o/oauth2/token", "client_id": "7508-5d3.apps.googleusercontent.com", "refresh_token": "1/Ux0k2DAjHYb6jbA" }

Now we can exit the container and re-launch it to run the ok-google sample and pass in all of the variables we have discovered.

MIC_ADDR="hw:2,0" SPEAKER_ADDR="hw:2,0" PROJECT_ID=voice-kit-01 MODEL_ID=voice-kit-01-number2-flc5yj CLIENT_ID=7508-5d3.apps.googleusercontent.com CLIENT_SECRET=kF4ZLu-X REFRESH_TOKEN=1/Ux0k2DAjHYb6jbA docker run -it --rm --device /dev/snd
-e MIC_ADDR=$MIC_ADDR -e SPEAKER_ADDR=$SPEAKER_ADDR -e PROJECT_ID=$PROJECT_ID
-e MODEL_ID=$MODEL_ID -e CLIENT_SECRET=$CLIENT_SECRET -e CLIENT_ID=$CLIENT_ID
-e REFRESH_TOKEN=$REFRESH_TOKEN
--name ok-google-cli-demo opensourcefoundries/ok-google setup authorized credentials file setup alsa configuration + exec bash -c 'googlesamples-assistant-hotword --project_id ${PROJECT_ID} --device_model_id ${MODEL_ID}' device_model_id: voice-kit-01-number2-flc5yj device_id: FB5DB23

https://embeddedassistant.googleapis.com/v1alpha2/projects/voice-kit-01/devices/FB5DB23 404 Registering.... Device registered. ON_MUTED_CHANGED: {'is_muted': False} ON_START_FINISHED

ON_CONVERSATION_TURN_STARTED ON_END_OF_UTTERANCE ON_RECOGNIZING_SPEECH_FINISHED: {'text': 'what time is it'} ON_RESPONDING_STARTED: {'is_error_response': False} ON_RESPONDING_FINISHED ON_CONVERSATION_TURN_FINISHED: {'with_follow_on_turn': False}