Table of Contents:

This document aims as a single resource for all information related to Apertis Release Process. It covers the Major Release process as well as the Point Release process.

Apertis Infrastructure

The Apertis project is hosted on a couple of infrastructure services, which are tightly coupled to each other, providing end-to-end automation. This includes:

  • Landing new packages into Apertis
  • Landing new security and general updates into Apertis
  • Landing downstream changes into Apertis

This document stands as a comprehensive guide to all package related processes as outlined above.

At a higher level, Apertis infrastructure includes:

  • Gitlab instance for source code hosting, CI, Automation, Image building etc
  • OBS (Open Build Service) for package building
  • reprepro for APT repository management
  • images.apertis.org for Apertis image hosting
  • LAVA for Quality Assurance Testing

Release Preparation

Below are a list of tasks that are usually required to be run before planning the execution of a release.

  • Keep release schedule wiki page updated
  • Announce soft feature freeze
  • Full archive rebuild (in the rebuild repository, not on the main one) to ensure all the packages are buildable from sources. See the the Archive Rebuilds section below
  • Announce hard feature freeze
  • Update the freshness status of all the design documents
  • Draft the release notes
  • Draft the internal release notes

Archive Rebuilds

Archive Rebuilds are done before a release to ensure that the entire repository of packages are in good shape, can build proper, and that all their inter-dependencies are satisfied. The recommended timeline for an archive-wide rebuild is right after the Feature Freeze. Once a freeze is in place, we don’t anticipate new changes to be introduced, and thus this is the optimal time to re-build all the packages slated for the release.

Note: An archive-wide rebuild is a time consuming task. Depending on the build infrastructure, it can take large amount of time. Thus, it is important to account for this time and plan a rebuild accordingly to ensure we have proper build images for the first Release Candidate, which is typicall 2 weeks from the Feature Freeze date.

Some notes about the Archive Rebuild task

  • Rebuilds are usually not required to be triggered manually.
  • Archive wide rebuild of packages should never be triggered on the main repositories. Use separate rebuild repositories and delete them once the rebuilds complete:
  • Create the rebuild repositories. Use automation script clone-rebuilds.sh from the apertis-infrastructure repository.
  • Ensure that all rebuild repositories have rebuild="local" block="never" in the project meta configuraiton. This is an important step needed to ensure that packages don’t get into cyclic rebuilds everytime a relevant library is built. 
       <repository name="rebuild" rebuild="local" block="never">
     <path project="apertis:v2021dev3:target" repository="default"/>
     <arch>x86_64</arch>
     <arch>armv7hl</arch>
     <arch>aarch64</arch>
   </repository>
  • Trigger the rebuilds: osc rebuild apertis:v2019:target --all -r rebuild
  • Delete rebuild repositories
  • Never trigger an archive wide rebuild on the primary repositories like target, sdk, development, hmi, helper-libs etc.

Release Branching

Once the Release Preparation is complete, the next step in the Release Process is to run the Release Branching steps.

At a higher level, Release Branching includes the following steps which needs to be run in close co-ordination.

  • Branching all package repositories with the new release
  • Branching all image recipes
  • Branching all test recipes
  • Branching docker image recipes
  • Branching all OBS repositories

Given the number of steps involved in Release Branching, the following Branching Checklists should be executed in close co-ordination.

Branching Checklists

There are a couple of checks to be performed on individual host machines. The machine names mentioned are specific and should be adopted to users' environment

  • Host niobium: Hosts the OBS service
  • Host images.apertis.org: Hosts the Apertis images repository
  • Host gitlab.apertis.org: Hosts the Apertis Gitlab instance

Branching Preparation

  • set up ~/.ssh/config such that the niobium alias is available with appropriate root privilege.
Host niobium
    User andrewshR

Host images.apertis.org
    User andrewshR
  • Check available disk space on OBS Server
    • Release Branching is a task that will consume a significant amount of disk space. Thus, it is very important to ensure that enough free space is available before commencing a release branching. Depending upon the size of a project, the amount of disk space required can vary.
    • Running out of disk space, halfway during the release branching process, may have severe side-effects. Hence, it is advised to consider over provision of free disk space before initiating a release branching.
  • Install the jq package. It is already packaged in Apertis
  • Install the osc-plugin-clone plugin for the OBS client
  • Install the osc-plugins-dput. It is already packaged in Apertis
  • Create a GitLab personal access token and configure it in $HOME/.gitlab-apertis-org-token, e.g. GITLAB_TOKEN=XXXXxxxxxXXXXXX > $HOME/.gitlab-apertis-org-token
    • Make sure your API token is active. When you create your API token, you have an option to set an expiry date. So make sure to verify the validity of your granted token
  • Ensure that the osc tool is configured with the Admin user credentials for OBS. This is essential for the creation of additional repositories
  • Check out the gitlab-rulez repository and unprotect the current release branch for all repositories.
    • Check out the repository
    • Verify that all relevant package groups, like pkg/*, have push_access_level: DEVELOPER_ACCESS in rulez.yaml
    • Apply the rule with ./gitlab-rulez apply rulez.yaml
  • Check out the apertis-infrastructure repository
  • Change the working directory to release-scripts/
  • Launch export RELEASE=v2019; export NEXT_RELEASE=v2020dev0; ./do-branching.sh which will:
    • Set new release to string v2020dev0 and set current release to string v2019, for example.
    • Create a new debootstrap script entry on the image builder host
    • Update the debootstrap package on OBS
    • Update the apertis-customizations, apertis-image-customization, apertist-tests repositories, both public and internal
    • Create branches in every repository on gitlab.apertis.org
    • Stop the OBS schedulers
    • Fork the OBS projects, copying packages and metadata
    • Create a new reprepro repository forcing a re-export of the binaries
    • Restart OBS to pick up the new reprepro entry (the magic is in /etc/obs/BSConfig.apertis.pm)
    • Clone and republish the OBS binary repositories
    • Restart the OBS schedulers
  • Update build suffix on OBS Project configuration (eg. bv2019.0b<B_CNT>, bv2020dev0b<B_CNT>). Please refer to section Build Suffix for more details about it
  • Add a directory for the release to the LAVA NFS path at: /srv/images/public/lava/nfs/ on host images.apertis.org. The LAVA test jobs pick the kernel and initrd image from the release’s directory path. Refer to the README for details
  • Instantiate a new timer to rotate images for the new release (eg. On host aura, run command systemctl enable --now rotate-images@v2019.timer)
  • Update the ci-package-builder to target the next release
  • Update the gitlab-rulez recipe to point to the new development release
    • Set the default development branch for all git projects
    • Set the protection settings onto all apertis/* branches, which were unprotected in the initial release steps
  • Manually branch the base-files package, after the new docker images have been built.
    • This step should automatically update the release specific attributes in the new package and auto propose the changes through a new merge request

Build Suffix

For every release (Developer, Preview, Product), we add a build suffix string to the packages, which relates to the release name of Apertis. The build suffix gets added to every built .deb package. Having a build suffix helps determine which Apertis release the .deb package was built for.

The suffix string is constructed of: b + Release Name + b<B_CNT>

  • Where, the initial b is for backward compatibility ensuring the new suffix string can work well together with older release packages
  • String Release Name will refer to the Apertis release’s name
  • String, b<B_CNT> refers to being a binary build along with the build count

For example, for an Apertis Developer Release of v2020dev0, we add string bv2020dev0b<B_CNT> to the project configuration on OBS. Similarly, for an Apertis Product Release of v2019.0, we add string bv2019.0b<B_CNT> to the project configuration on OBS

Stable Build Suffix

Ensure to update the build suffix on the project configs when doing a Stable Point Release

For Point Releases (Eg. v2019.1, v2019.2), we need to update the build suffix to ensure that the packages in the new point release have a higher version than the one in the updates or security repository.

For example, during a release, the build suffix for a v2019 release is: bv2019.0b<B_CNT>. The same build suffix is inherited by the :updates and :security sub-repository. Before doing a new point release v2019.1, we need to set the build suffix of the main repositories (apertis:v2019:target, apertis:v2019:development, apertis:v2019:sdk, apertis:v2019:hmi etc) to vb2019.1b<B_CNT>.

This will ensure that the packages synced from the :updates and :security sub-repository will have a higher version, when they are synced to the main repository

Stable Point Release

Point Releases for the Stable Release happen on a timely cadence as defined in the Release Schedule. A Stable Point Release includes:

  • Security Updates to software packages (funneled through $RELEASE-security). These include security vulnerability fixes.
  • Normal updates to software packages (funneled through $RELEASE-updates). These include bug fixes to existing version of software packages.
  • Backport of software packages (funneled through $RELEASE-backports). These include newer version of software packages backported to older OS releases

Stable Point Release Preparation

  • Increment the Build Suffix in the Project configuration for each repository relevant to the release. See section Build Suffix for more details
  • Run the fold-security-update-branches script to:
    • List the packages in the :security, :updates and other similar repositories
    • Fast Forward Merge --ff-only the respective security and updates branches on Git. For example, apertis/v2019-security and apertis/v2019-updates should be merged back to apertis/v2019. Note: Not able to perform a fast forward merge means something got broken and needs to be investigated and fixed.
    • Poke the pipeline and ensure that all the new packages (from the merged back changes to apertis/v2019) get built on OBS and land into the main repository.
    • Drop the updates, security branches (Eg. apertis/v2019-updates, apertis/v2019-security)
    • Delete the packages from the updates, security etc repositories on OBS (Eg. apertis:v2019:updates:target, apertis:v2019:security:development)
    • Delete the pacakges from the reprepro database
  • Manually validate that the fold-security-update-branches script has merged all the updates into main repository
  • Rebuild the image to pick up the updated packages in the main repository.
  • Run the copy-release script on the images server (aura) to prepare the release.
  • copy-release -v 20191120.0 v2019.1rc2
  • copy-release -v -i 20191120.0 v2019.1rc2
  • First for the public images and then second for Internal images (-i option).
  • The source folder is picked from the daily image builds.
  • Send release announcement for the release.

Stable Point Release - Automated Steps

Launch fold-security-update-branches script from the apertis-infrastructure/release-scripts/gitlab/, which automates many of the steps in a Stable Point Release Preparation

  • List the packages in the :security, :updates and other similar repositories
  • Fast Forward Merge --ff-only the respective security and updates branches on Git. For example, apertis/v2020-security and apertis/v2020-updates should be merged back to apertis/v2020. Note: Not able to perform a fast forward merge means something got broken and needs to be investigated and fixed.
  • Poke the pipeline and ensure that all the new packages (from the merged back changes to apertis/v2020) get built on OBS and land into the main repository.
  • Drop the updates, security branches (Eg. apertis/v2020-updates, apertis/v2020-security)
  • Delete the packages from the updates, security etc repositories on OBS (Eg. apertis:v2020:updates:target, apertis:v2020:security:development)
  • Delete the pacakges from the reprepro database

Following are the example runs for the automation script.

Example Run with help sub-command

rrs@priyasi:~/$ ./fold-security-update-branches  help
Usage: ./fold-security-update-branches [apply] [gen-pkg-list] [dry-run] [help]

Example Run with gen-pkg-list sub-command

The gen-pkg-list sub-command generates a bunch of data files, based on the Apertis Release and Apertis Repositories. These generated data files are used by the script to process changes (Security and Regular Updates) in Apertis point releases.

rrs@priyasi:~/rrs-home/devel/CCU/Apertis/apertis-infrastructure/release-scripts/gitlab (v2020-folding-work-data)$ ./fold-security-update-branches  gen-pkg-list
Package lists extracted from OBS' repository v2020-security-target.txt
Package lists extracted from OBS' repository v2020-updates-target.txt
Package lists extracted from OBS' repository v2020-security-development.txt
Package lists extracted from OBS' repository v2020-updates-development.txt
Package lists extracted from OBS' repository v2020-security-sdk.txt
Package lists extracted from OBS' repository v2020-updates-sdk.txt
Package lists extracted from OBS' repository v2020-security-hmi.txt
Package lists extracted from OBS' repository v2020-updates-hmi.txt

Example Run with dry-run sub-command

The dry-run sub-command will run the actual commands without any effect on the server-side. This is very handy to check and determine the actual actions that will be run.

$ ./fold-security-update-branches  dry-run
Package apt is in sync in branches: v2020 and v2020-security
Deleting branch v2020-security
Package bluez is in sync in branches: v2020 and v2020-security
Deleting branch v2020-security
Package boost1.67 is in sync in branches: v2020 and v2020-security
Deleting branch v2020-security
Package cups is in sync in branches: v2020 and v2020-security
Deleting branch v2020-security

Example Run with apply sub-command

The apply sub-command does the actual job.

$ ./fold-security-update-branches apply
Running in apply mode
Package apt is NOT IN SYNC in branches: v2020 and v2020-security
Merging branch v2020-security into v2020
https://gitlab.apertis.org/pkg/apt/-/merge_requests/10
Package bluez is NOT IN SYNC in branches: v2020 and v2020-security
Merging branch v2020-security into v2020
https://gitlab.apertis.org/pkg/bluez/-/merge_requests/12
Package boost1.67 is NOT IN SYNC in branches: v2020 and v2020-security
Merging branch v2020-security into v2020
https://gitlab.apertis.org/pkg/boost1.67/-/merge_requests/5
Package cups is NOT IN SYNC in branches: v2020 and v2020-security
Merging branch v2020-security into v2020
https://gitlab.apertis.org/pkg/cups/-/merge_requests/8
Package curl is NOT IN SYNC in branches: v2020 and v2020-security
Merging branch v2020-security into v2020
https://gitlab.apertis.org/pkg/curl/-/merge_requests/7
Package cyrus-sasl2 is NOT IN SYNC in branches: v2020 and v2020-security
Merging branch v2020-security into v2020

Removing an old distribution

Perform the following steps to remove an old distribtuion. This would usually be the obsolete Beta and Preview distribution releases, like: v2019dev0, v2019pre

  • Remove the entry for the distribution from the reprepro configuration file. Example: /srv/obs/repos/shared/apertis/internal/apertis/conf/distributions

  • Run the following reprepro command to apply the changes

    # sudo -u obsrun reprepro --gnupghome /srv/obs/gnupg/ -Vb /srv/obs/repos/shared/apertis/internal/apertis --delete clearvanished
Deleting vanished identifier 'u|v2019dev0|proprietary|amd64'.
Deleting vanished identifier 'u|v2019dev0|proprietary|arm64'.
Deleting vanished identifier 'u|v2019dev0|proprietary|armhf'.
Deleting vanished identifier 'u|v2019pre|demo|amd64'.
Deleting vanished identifier 'u|v2019pre|demo|armhf'.
Deleting vanished identifier 'u|v2019pre|nothumb|amd64'.
Deleting vanished identifier 'u|v2019pre|nothumb|armhf'.
Deleting vanished identifier 'u|v2019pre|proprietary|amd64'.
Deleting vanished identifier 'u|v2019pre|proprietary|armhf'.

Appendix

Helper scripts to ease Apertis development

release-scripts/: contains helper scripts to ease release process.

  • update-* scripts create branches in key Git repositories for the current release and commit changes necessary to build images for the next release.
  • create-branches is a script to create branches for the current release in all Git repositories. This script requires curl and jq.
  • add-new-repo is a script which updates the reprepro configuration at the OBS backend host, and it has to run there as root. It depends on OSC clone plugin.
  • do-branching.sh is the main script used to branch the next release off in preparation for the current release.
    • In addition to RELEASE and NEXT_RELEASE it also reacts on NOACT skipping the execution of most commands when set to any non-empty value. NOVERIFY (currently hardcoded to ‘y’) skips the verification of the package copy process.
    • NOTE: The branching script expects root access on certain hosts to be configured in ~/.ssh/config