Fiotest - part II

Part II of the fiotest series series will focus on some examples of what you can do with fiotest. Hopefully you'll find some inspiration and things you can copy/paste into your own tests.

This article includes examples to demonstrate:

• Uploading artifacts from tests
• Adding test-result details to a test
• Testing status of another container running on the host
• Including metrics in test results
• Running tests on the host LmP
• Creating a custom container with a new test

Uploading artifacts from tests

Fiotest includes a test artifact containing a test's stdout/stderr console.log. However, you can also include other data you may need for test results or diagnostics.

test-spec.yml

sequence:
- tests:
  - name: upload-artifact
    command:
    - /tests/test-artifact.sh

tests/test-artifact.sh

#!/bin/sh
echo "testing upload artifact"
/usr/bin/dmesg > $ARTIFACTS_DIR/dmesg

Adding test-result details to a test

Sometimes you may have something more like a test suite that includes individual tests. Fiotest calls these "test-results". It allows you to do something like:

test-spec.yml

sequence:
- tests:
  - name: test-with-results
    command:
    - /tests/test-with-results.sh

tests/test-with-results.sh

#!/bin/sh -e

echo "This is an example of a test that includes test-results for fine grained reporting"
fio-test-result test-that-fails /bin/false
fio-test-result test-that-passes /bin/true

That will produce something like:

$ fioctl targets tests 191 02961acb-aed0-4b32-8077-9c3808db16ce
Name:      test-results-example
Status:    FAILED
Created:   2020-11-20 20:18:08 +0000 UTC
Completed:
Device:    andy-test
Artifacts:
     console.log

TEST RESULT         STATUS
-----------         ------
test-that-fails     FAILED
test-that-passes    PASSED

Testing status of another container running on the host

Since most work we recommend users do takes place in other containers, it's useful to be able to communicate with them. This example shows how you could verify the "shellhttpd" example application is running:

test-spec.yml

sequence:
- tests:
  - name: shellhttpd
    command:
    - /tests/check-shellhttpd

tests/check-shellhttpd

#!/usr/bin/python3
import netifaces
import requests

# shellhttpd binds port 8080 on the host's default interface.
# this is a quick way to find it:
host_ip = netifaces.gateways()["default"][netifaces.AF_INET][0]
print("Host IP:", host_ip)
r = requests.get(f"http://{host_ip}:8080")
r.raise_for_status()

Including metrics in test results

Tests results can include metrics. Metrics are dictionary of label(type string)->value(type float). This can be done by making a small addition to the previous check-shellhttpd test:

tests/check-shellhttpd

#!/usr/bin/python3
import os

import netifaces
import requests

host_ip = netifaces.gateways()["default"][netifaces.AF_INET][0]
print("Host IP:", host_ip)
r = requests.get(f"http://{host_ip}:8080")
r.raise_for_status()

# fio-test-result will look for things in $METRICS_DIR to upload to server
with open(os.path.join(os.environ["METRICS_DIR"], "foos_per_second"), "w") as f:
    f.write("%f" % 42.1)

Fioctl doesn't currently display metrics. However, the REST API does expose this information:

curl -H “OSF-TOKEN: <API TOKEN>” https://api.foundries.io/ota/factories/andy-corp/targets/191/testing/fe6ffaa3-d263-41ce-bb2f-627747f76cb3/

{
   "artifacts" : [
      "console.log"
   ],
   "completed-on" : null,
   "created-on" : 1605905297,
   "details" : "",
   "device-name" : "andy-test",
   "device-uuid" : "c9207945-0687-43ab-a505-3079fef300c9",
   "name" : "test-results-example",
   "results" : [
      {
         "details" : "",
         "name" : "test-that-fails",
         "status" : "FAILED"
      },
      {
         "details" : "",
         "name" : "test-that-passes",
         "status" : "PASSED"
      },
      {
         "details" : "",
         "metrics" : {
            "foos_per_second" : 42.1
         },
         "name" : "shellhttpd-running",
         "status" : "PASSED"
      }
   ],
   "status" : "FAILED",
   "test-id" : "abfd2e56-6056-4bcb-a30e-2957c9e8c750"
}

Running tests on the host LmP

We pride ourselves in demonstrating how much you can do in a container. However, testing often needs to be done to prove out the host OS as well. We've made this easy.

test-spec.yml

sequence:
  - tests:
    - name: system
      on_host: true
      command:
        - /home/fio/host-tests.sh

The on_host: true instructs fiotest to ssh into the host and run the given command. The path to the command must exist on the host. We sometimes (ab)use this functionality to run device specific testing on our equipment.

Creating a custom container with a new test

Customers often need to add some custom packages to the fiotest container that includes test dependencies. This example shows how to include the stressng test:

Dockerfile

FROM hub.foundries.io/lmp/fiotest:postmerge

RUN apt update && \
    apt install -y stress-ng && \
    rm -rf /var/lib/apt/lists/*

docker-compose.yml

# this file stays the same except for fiotest.image:
image: hub.foundries.io/<your factory>/fiotest:latest

test-spec.yml

sequence:
- tests:
  - name: stress-ng
    command:
    - /usr/bin/stress-ng
    - --cpu=4
    - --io=2
    - --vm=2
    - --timeout=10s

What's next

The next article will cover two topics in one blog. I'll cover how I develop fiotest. This should not only help show how you can extend fiotest, but also show some tricks I use for developing applications within my personal Factory.