Personal LAVA Tests

This tutorial explains how to submit a personal LAVA job for an Apertis test, which is very useful during development, either to debug a test, or to check that everything is working as expected with the test before its final integration.

Running a personal test basically consists in adding the LAVA test definition file in a personal repository and submitting a LAVA job that will fetch this file from there to execute the test.

Install LQA

lqa is the tool used to submit the daily automated LAVA jobs for the Apertis images, so it will also be used to submit the Apertis job in this tutorial. Before running the instructions in this tutorial, please check the following link to install and configure lqa and create a personal bundle stream: LQA

Package dependencies

This tutorial assumes that all the packages dependencies required by the personal test are already available in the Apertis packages repositories, if that is not the case, any missing package dependency should ideally be first integrated into the packages repositories.

Nevertheless, there are ways to fetch external packages from LAVA jobs. The two main ways are:

Add a new test definition to include the package repository where the package resides during the test execution (check the file apertis-tests/misc/add-repo.yaml for an example).

Or:

Fetch the package directly with some command line utility from the test definition itself and install it manually during the test execution.

Test definition

The test definition is the YAML file containing the steps to execute the test in LAVA. This file can be fetched by LAVA from different kind of locations, in the Apertis project these files are available from git repositories.

So, once you write the test definition file, you need to make available this file from your own personal git repository from which LAVA can fetch it.

Job definition

The job definition is a YAML file that tells LAVA how to execute the test job. This file will point LAVA to fetch the test definition from the personal git repository.

apertis-tests Git repository comes with a set of templates which simplify the job of creating the job definition. The common-boot-tpl.yaml and common-qemu-boot-tpl.yaml templates define device-specific actions to boot an image on a real device or in QEMU. The job definition using these templates thus can be dramatically simplified.

The following is a YAML job template using the abovementioned common code to run a simple test:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

job_name: lsdisk test on {{release_version}} {{pretty}} {{image_date}}
{% if device_type == 'qemu' %}
{% include 'common-qemu-boot-tpl.yaml' %}
{% else %}
{% include 'common-boot-tpl.yaml' %}
{% endif %}

  - test:
      timeout:
        minutes: 5
      namespace: flash
      name: lsdisk
      definitions:
        - repository: https://git.apertis.org/git/apertis-tests.git
          revision: master
          from: git
          path: misc/lsdisk.yaml
          name: lsdisk

Each test is listed as a separate item under definitions. The test names should be unique. It is also possible to add another test section, if the separation makes the test set more readable to use. It makes no difference at all on how the tests are run or the results are presented.

Save the above template as a file named my-test-tpl.yaml.

Since most of the fields in the template should work just fine by default, to tweak values from the common code it is required to edit them directly. You might need to change the timeouts values if the test can take a long time.

Choose Apertis profile (image type)

The apertis-tests repository contains most of the Apertis tests files and all the required files to execute the automated tests in LAVA.

There are several types of Apertis images, and there exist a profile for each of them that can be used by lqa to execute that specific image type.

Fetch the apertis-tests repository to get the Apertis profile file:

$ git clone https://git.apertis.org/git/apertis-tests.git

Check the file templates/profiles.yaml to select the profile you want to run, or just run the following the command to get all the profiles names from that file directly:

$ grep name apertis-tests/templates/profiles.yaml

Each profile specifies an Apertis image type, containing all the specific values for it.

The next step is just to submit the job to LAVA using the selected profile.

Submit LAVA job

Triggering a LAVA job is just submitting the YAML job file with the required options using the lqa command.

As an example, to trigger a job to execute the above example template saved as my-test-tpl.yaml on QEMU for the Apertis Target AMD64 image with date 20171125.0 the following command can be used:

$ lqa submit -g apertis-tests/templates/profiles.yaml \
        -p apertis-17.12-daily-qemu-target \
        -t image_date:20161025.0 my-test-tpl.yaml

The -g option points to the profile file from the apertis-tests repository, and the -p option selects the profile we want to use.

The -t option lets users set or override variables used to parametrise the templates.

You need to be in a directory containing apertis-tests to be able to execute the command above.

lqa submit has many options to submit jobs, for example, the combination of the -n and -v options are very useful to check for the final generated JSON file before submitting to the server. Check the complete options with: lqa submit --help.

Once the command is executed, there should be a LAVA job running for this personal test at: https://lava.collabora.co.uk/

If a job template file is not specified from the command line, it will trigger lava jobs for all the templates found in the profiles.yaml file for the specific profile.

Details on test job templates

The boot process for non-emulated devices and for QEMU differs, and due to the amount of differences the definitions are split into two separate template files.

common-boot-tpl.yaml contains definition needed to boot Apertis images on real (non-emulated devices). Since they cannot boot images directly, the boot process is separated in two stages: flashing the image onto a device from which the board can boot, and booting into the image and running tests.

The first stage boots over NFS into a (currently) Debian stretch image with a few extra tools needed to flash the image, downloads the image using HTTP, flashes it and reboots. This stage is defined using namespace: flash in the job YAML file. In most cases you won’t need to edit bits related to this stage. The second stage is common for both non-emulated devices and QEMU, despite them having certain differences. It is used to boot the image itself, prepare the LAVA test runner and run tests. This stage is defined using namespace: system. You normally don’t need to edit this stage either. The exception to this is when you need to load an image from a different source than images.apertis.org.

Image URLs are defined in the deploy action. For common-boot-tpl.yaml, it is necessary to specify URLs to both image itself and its bmap file, which is used to speed up the flashing process and avoid unnecessary excessive device wear. For common-qemu-boot-tpl.yaml, only the URL to the image itself is needed, as QEMU doesn’t support bmap files yet.

The second stage always performs two tests: sanity-check, which basically checks that the system actually works, and add-repo, which isn’t actually a test, and is used to add repositories to /etc/apt/sources.list on certain devices.

Non-public jobs

These instructions are written to submit LAVA jobs for ONLY PUBLIC Apertis images. If you need to submit a LAVA job for a private image, there are few things that need to be taken into consideration and few changes need to be made to these instructions: personal or group visibility should be selected for your jobs.