Guide to development within LAVA¶

Dependencies required to run unit tests¶

To run the unit tests, a range of dependencies need to be installed:

$ sudo apt -y install lava-dev

To reduce the number of unit tests which are skipped, a number of other packages are listed as recommended by lava-dev and these should be installed as well:

$ sudo apt -y install lxc u-boot-tools tftpd-hpa telnet nfs-kernel-server img2simg simg2img

Important

For security reasons, each installation of lava-server sets the permissions of /etc/lava-server/instance.conf to only be readable by the lavaserver user and users in the lavaserver group.

(Some older instances may have a different username for this user and group - check the value of LAVA_DB_USER in /etc/lava-server/instance.conf using sudo.)

To run the unit tests, the user running the unit tests needs to be in the lavaserver group. For example:

$ sudo adduser <username> lavaserver

Developing using device-type templates¶

If you are an administrator, you may think the previous link sent you to the wrong section. However, administrators need to understand how device-type templates operate and how the template engine will use the template to be able to make changes.

Important

Device type templates are more than configuration files - the templates are processed as source code at runtime. Anyone making changes to a device-type .jinja2 template file must understand the basics of how to test templates using the same tools as developers.

Device type templates as code¶

Jinja2 provides a powerful templating engine. Templates in LAVA use several standard programming concepts:

Conditional Logic
Inheritance
Default values

In addition, LAVA templates need to always render to valid YAML. It is this YAML which is sent to the worker as device.yaml. The worker does not handle the templates. All operations are done on the master.

Testing new device-type templates¶

The simplest check is to render the new template to YAML and check that it contains the expected commands. As with test job files, there are common YAML errors which can block the use of new templates.

See also

YAML syntax errors

lava-server manage device-dictionary --hostname <HOSTNAME> --review

All templates are checked for basic syntax and output using:

$ python3 -m unittest -vcf tests.lava_scheduler_app.test_base_templates.TestBaseTemplates.test_all_templates

A more rigorous test is to use the dedicated unit test which does not require lava-server to be installed, i.e. it does not require a database to be configured. This test can be run directly from a git clone of lava-server with a few basic python packages installed (including python-jinja2).

Individual templates have their own unit tests to test for specific elements of the rendered device configuration.

The number of unit tests and templates has increased, so there are dedicated unit test files for particular types of template unit tests:

$ python3 -m unittest -vcf tests.lava_scheduler_app.test_fastboot_templates

$ python3 -m unittest -vcf tests.lava_scheduler_app.test_grub_templates

$ python3 -m unittest -vcf tests.lava_scheduler_app.test_qemu_templates

$ python3 -m unittest -vcf tests.lava_scheduler_app.test_uboot_templates

Most changes to device-type templates take effect immediately - as soon as the file is changed in /etc/lava-server/dispatcher-config/device-types/ the next testjob for that device-type will use the output of that template. Always test your templates locally before deploying the template to the master. (Test jobs which have already started are not affected by template changes.)

Use version-control for device-type templates¶

This cannot be stressed enough. ALL admins need to keep device-type templates in some form of version control. The template files are code and admins will need to be able to upgrade templates when packages are upgraded and when devices need to implement new support.

Contribute device-type templates back upstream¶

As code, device-type templates need to develop alongside the rest of the codebase. The best way to maintain support is to Contributing Upstream so that new features can be tested against your templates and new releases can automatically include updates to your templates.

Some individual device files exist in the codebase in lava_scheduler_app/tests/devices but these are only for use in the existing unit tests. There is no need to contribute individual device dictionaries unless there are new unit tests which use those device dictionaries.

Developer workflows¶

Note

LAVA is developed using Debian packaging to ensure that daemons and system-wide configuration is correctly updated with changes in the codebase. There is no official support for pypi or python virtual environments or installing directly from a git directory. python-setuptools is used but only with sdist to create the tarballs to be used for the Debian packaging, not for install. Some dependencies of LAVA are not available with pypi, for example python-guestfs would need installing using another method.

See also

Developing LAVA on Debian and a summary of the Debian LAVA team activity

Developers can update the installed code on their own systems manually (by copying files into the system paths) and/or use symlinks where appropriate but changes need to be tested in a system which is deployed using the Developer package build before being proposed for review. All changes must also pass all the unit tests, unless those tests are already allowed to be skipped using unittest decorators.

Mixing the use of python code in /usr/local/lib and /usr/lib on a single system is known to cause spurious errors and will only waste your development time. Be very careful when copying files and when using symlinks. If in doubt, remove /usr/local/lib/python* and ~/.local/lib/python* then build a local developer package and install it.

If your change introduces a dependency on a new python module, always ensure that this module is available in Debian by searching the Debian package lists. If the module exists but is not in the current stable release of Debian, it can be backported but be aware that this will delay testing and acceptance of your change. It is expressly not acceptable to add a dependency on a python module which is only available using pypi or pip install. Introducing such a module to Debian can involve a large amount of work - talk to us before spending time on code which relies on such modules or which relies on newer versions of the modules than are currently available in Debian testing.

The dependencies required by LAVA are tracked using the ./share/requires.py script which is also available in the lava-dev package as /usr/share/lava-server/requires.py. Merge requests which need extra modules which already exist in Debian can add the relevant information to the share/requirements/debian files.

Note

For the CI to pass, the extra module(s) must be available for stable and testing. Pay particular attention to the version available in buster and buster-backports. If a minimum version is required, this can be specified in the requirements, as long as that version or newer is available in buster-backports. talk to us if your change involves new files that may need changes in the packaging code. All CI tests must pass before any new code can be merged, including building working packages containing the new support.

Code locations¶

All the code for the lava-server and lava-dispatcher support exists in the single LAVA repository:

https://gitlab.com/lava/lava

Includes:

lava_scheduler_app
lava_results_app
lava_server
lava
lava_common
linaro_django_xmlrpc
lava_dispatcher
lava_test_shell
debian

See also

Developing new classes for LAVA V2

Compatibility¶

See also

Compatibility failures

The compatibility mechanism allows the lava-master daemon to prevent issues that would arise if the worker is running older software. A job with a lower compatibility may fail much, much later but this allows the job to fail early. In future, support is to be added for re-queuing such jobs.

Developers need to take note that in the code, compatibility should reflect the removal of support for particular elements, similar to handling a SONAME when developing in C. When parts of the submission YAML are changed to no longer support fields previously used, then the compatibility of the associated strategy class must be raised to one more than the current highest compatibility in the lava-dispatcher codebase. Compatibility does not need to be changed when adding new classes or functionality. It remains a task for the admins to ensure that the code is updated when new functionality is to be used on a worker as this typically involves adding devices and other hardware.

Compatibility is calculated for each pipeline during parsing. Only if the pipeline uses classes with the higher compatibility will the master prevent the test job from executing. Therefore, test jobs using code which has not had a compatibility change will continue to execute even if the worker is running older software. Compatibility is not a guarantee that all workers are running latest code, it exists to let jobs fail early when those specific jobs would attempt to execute a code path which has been removed in the updated code.

Jinja2 support¶

The Jinja2 templates can be found in lava_scheduler_app/tests/device-types in the lava-server codebase. The reason for this is that all template changes are checked in the unit-tests. When the package is installed, the device-types directory is installed into /etc/lava-server/dispatcher-config/device-types/. The contents of lava_scheduler_app/tests/devices is ignored by the packaging, these files exist solely to support the unit tests.

See also

Unit-tests and Testing the design for examples of how to run individual unit tests or all unit tests within a class or module.

Device dictionaries¶

Individual instances will each have their own locations for the device dictionaries of real devices. To allow the unit tests to run, some device dictionaries are exported into lava_scheduler_app/tests/devices but there is no guarantee that any of these would work with any real devices, even of the declared device-type.

For example, the Cambridge lab stores each device dictionary in git at https://git.linaro.org/lava/lava-lab.git and you can look at the configuration of staging as a reference: https://git.linaro.org/lava/lava-lab.git/tree/staging.validation.linaro.org/master-configs/staging-master.lavalab/lava-server/dispatcher-config/devices

Device dictionaries can also be downloaded from any LAVA instance using the XML-RPC call, without needing authentication:

server.scheduler.devices.get_dictionary(hostname)

Dispatcher device configurations¶

The lava-dispatcher codebase also has local device configuration files in order to support the dispatcher unit tests. These are not Jinja2 format, these are YAML - the same YAML as would be sent to the dispatcher by the relevant master after rendering the Jinja2 templates on that master. There is no guarantee that any of the device-type or device configurations in the lava-dispatcher codebase would work with any real devices, even of the declared device-type.

Contributing Upstream¶

The best way to protect your investment on LAVA is to contribute your changes back. This way you don’t have to maintain the changes you need by yourself, and you don’t run the risk of LAVA changed in a way that is incompatible with your changes.

Upstream uses Debian, see Developing LAVA on Debian for more information.

Planning¶

The LAVA Software Community Project uses GitLab for all development and planning for new features and concepts. Discussion happens on the Mailing lists.

Many older git commit messages within the LAVA codebase contain references to JIRA issues as LAVA-123 etc., as the LAVA project used to use Linaro’s JIRA instance to track issues. All references like this can be appended to a basic URL to find the details of that issue: https://projects.linaro.org/browse/. e.g. the addition of this section on JIRA relates to LAVA-735 which can be viewed as https://projects.linaro.org/browse/LAVA-735

If you have comments or questions about anything visible within the LAVA project, please subscribe to one of the mailing lists and ask your questions there.

Report a Bug¶

The LAVA Software Community Project uses GitLab for all bugs, issues, feature requests, enhancements and problem reports. For more general questions and discussion, use the Mailing lists. It is often useful to discuss the full details of the problem on the lava-users mailing list before creating an issue in GitLab.

Note

The old Bugzilla and JIRA systems are both deprecated and reporting bugs in the old Bugzilla system will not be tracked by the LAVA team.

Community contributions¶

The LAVA software team use GitLab to manage contributions. For details, please read the LAVA Software Community Project Contribution Guide

See also

Release process and LAVA development and LAVA Software Community Project Contribution Guide for detailed information on running the unit tests and other static code analysis tools before submitting the review.

Contributing via your distribution¶

You are welcome to use the bug tracker of your chosen distribution. The maintainer for the packages in that distribution should have an account on https://gitlab.com/lava/lava to be able to forward bug reports and patches into the upstream LAVA systems.

See also

https://www.debian.org/Bugs/Reporting

Contributing via GitHub¶

GitHub has mirrors of the GitLab repository but merge requests need to be run through the GitLab CI tests. This can be done by changing the git remote of the GitHub branch and pushing to GitLab. GitHub users can create GitLab accounts on https://gitlab.com using their GitHub credentials.

See also

LAVA development.