Test Case Guidelines

The following are guidelines to create and edit test cases. Please follow them unless there is a compelling reason to make an exception.

Workflow

1. Before developing a new test case:
   • Make sure the test case doesn’t exist by checking the Test Cases list.
   • Make sure nobody is already working on a test case by checking the Phabricator tasks.
   • Determine the main feature to focus test upon and set the test case identifier accordingly.
2. While developing the test case:
   • Create the test case page at the wiki path by following the Test Case template.
   • Set the “state” parameter of the template to Devel or to if available.
   • Check QA/ImmutableRootfsTests in order to write tests that can be run properly on all systems.
3. Once the test case is finalized:
   • Ask for a review from QA team.
   • The reviewer will change the “state” parameter to Final once he’s satisfied.
4. When a test suite is modified:
   • The test case should be synced. There should be no disparity between the test case and its suite.
   • Edit the version history to reflect the changes made.
5. When a test is deleted or becomes deprecated:
   • Change the “state” parameter to Deprecated.
   • Edit the version history to make it clear the test case is no longer used to test newest releases.
   • If the test case relies on files that have been moved or edited, modify the steps to reflect these changes (e.g. checkout an old git commit where the needed script was still intact).
   • If the test case relies on files no longer available (e.g. online resource), add a warning on the page to that effect.

Identification

To keep test case identifiers consistent, they should be in the following form:

    <component>-<feature>-<sub-feature>-<suffix>

:* The component part is mandatory. It can be a module name or one of the following keywords: sdk, image or system.

:* The feature part is useful to mark the relation with the specification. It should be common between several test cases.

:* The sub-feature part should be concise but precise enough to describe the specific functions tested by that case.

:* The following suffixes can be used in specific scenarios:

::* unit - Unit test suite shipped with the component.

::* performance - Performance oriented test.

::* compatibility - Compatibility test.

Moreover, the following rules should be taken into account while choosing the identifier:

• All lower case characters.
• Use single words to describe components and features. If not possible, use dashes to separate words (e.g. hw-accelerated).
• Be concise yet thorough.
• Avoid using mutable states in names (e.g., foo-bar-manual, because it may later be automated).
• Avoid using only the component as identifier (e.g. libbar) or some generic suffix (e.g. foo-all). It usually means the test case should be split.

Here are some good identifier examples:

• folks-metacontacts-linking (component: folks, feature: metacontacts, sub-feature: linking)
• gettext-i18n (component: gettext, feature: i18n)
• sdk-roller-widget-performance (component: sdk, feature: roller-widget, suffix: performance)
• telepathy-gabble-unit (component: telepathy-gabble, suffix: unit)

Path

Each test case should be created at a wiki path of the following form:

    QA/Test_Cases/<test-case-name>

Example:

• QA/Test Cases/image-bootable

Template

The template Template:TestCase is the one to used to create new test cases. Please read thoroughly the documentation on that page before continuing here.

The template automatically adds the page to the QA/Test_Cases category, so all existing test cases are listed there.

Please follow these rules while using the template:

• Fill the description field appropriately. It should explain what the test case is testing even if it is fully automated.
  • Example of valid description: “Test that component A correctly supports feature 1 and feature 2 in condition X”.
  • Example of invalid description: “Run a test script for component A”.
• The description field should also contain a mandatory explanation about how the test is being executed and what exact feature is testing, for example:
  • “This test runs script X in this way, then run Y, and execute steps A-B-C to test feature Z”
  • More examples and explanation on template page.
• While listing pre-conditions and execution steps, make sure to include every single step needed when starting from a clean image. Don’t make any assumption.
  • List all packages needed that aren’t installed by default on the image.
  • List all necessary resources and how to retrieve them.
  • List all expected environment variables.
  • etc.

Helper Templates

Take advantage of MediaWiki’s templating system to factor out common patterns on pages. Here are the available templates that can be used when writing test cases.

These are organized roughly from most- to least-general.

Packages and Package Repositories

• Template:Install dependencies - Add steps to install dependencies needed for the execution of the test.

      {{Install dependencies|libfoo libbar}}
  

• Template:Enable package repositories - Add steps to add some package repositories (max. 4) containing dependencies.

      {{Enable package repositories|repo_1|repo_2|repo_3|repo_4}}
  

Installing and Configuring Images

• Template:Install sdk - Add steps to download, install and run the SDK image in VirtualBox.

      {{Install sdk}}
  

Download Self-Contained Test Data

• Template:DownloadTestData - Add steps to download self-contained test data for tests to be run on images and containers with an immutable rootfs where packages cannot be installed

      {{Install DownloadTestData}}
  

Bluetooth

• Template:EnableBluetooth - Add a warning about Bluetooth being disabled by default and steps to enable it.

      {{Enable bluetooth}}
  

• Template:PairBluetoothDevice

  • Add steps to pair the system with another Bluetooth device.

      {{Pair bluetooth device}}
    

Misc. Warnings

• Template:ConnmanPersistenceWarning

  • Add a warning about persistence of changes with Connman.

      {{Connman persistence warning}}