Skip to content

Testing modm

modm includes a large number of example as well as an extensive unit test suite which can be compiled and execute on hosted and embedded targets. We‘ve written our own small test harness to be able to fit the tests onto very small devices like AVRs. As part of our Continuous Integration checks, all examples are compiled and hundreds of unit tests are executed to help us catch regressions.

However, there is room for improvement in both our test coverage, as well as our test execution. For example, while most important platform-independent code is covered in our unit tests, the HAL is not tested systematically but only manually whenever a developer executes an example on their hardware or works on the HAL. Nevertheless, the quality is still fairly high, mostly due to the relative simplicity of our HAL driver code.

Here are some guidelines and tips on how to make the best of what testing abilities we have available today.

Manual Tests

There is plenty of opportunity to customize modm, either using lbuild‘s module options or simply by using the HAL on your specific hardware. This means the more examples we have of different configurations and use-cases the better we can debug existing problems and prevent new ones.

So when working on a fix or a new feature, make sure to create a simple example, so that others have an idea of the environment in which it was developed. Test your new code on real hardware with the compiler toolchain we use and make sure to document any caveats you discover.

You can compile all examples locally using our helper script.

 $ cd examples
 $ ../tools/scripts/examples_compile.sh

Add your example to our CI

If you created a new folder inside modm/examples, you need to add it to one of the jobs in .circleci/config.yml:

- run:
  name: Examples YOUR TARGET
  command: |
    (cd examples && ../tools/scripts/examples_compile.sh new_folder)

Unit Tests

Our unit tests are located in the modm/test directory and are generated using lbuild as well. They are all submodules of modm:test and are all platform independent. We‘ve written a small Makefile for the most commonly used commands.

 $ cd tests
   # generates, compiles and executes the unit test for hosted targets
 $ make test-hosted-linux
 $ make test-hosted-darwin
   # generates and compiles the unit test for embedded targets
 $ make test-stm32
 $ make test-avr

Adapt for your target

To execute the unit tests on your embedded targets, you may have to adapt the project files (avr.xml or stm32.xml) as well as the hardware setup (like modm/test/runner/stm32.cpp.in) for the target you have available.

The unit tests use the SCons build system, since that is the most tested. You can simply program the unit test onto your target by calling scons program inside the build directory (modm/build/unittest/{avr, stm32}). The results will be reported on the UART output you configured.

The unit test library we use is located in modm/src/unittest which is the modm:unittest modules. See the documentation for that modules for details on how to write your own unit tests.

Test all Targets

Apart from unit tests, we also generate the HAL (all modules in modm:platform:**) for ALL targets we support and compile it with a simple GPIO example. This doesn't catch everything, but it does make sure that these modules are at least compilable, which is particularly interesting when writing new peripheral drivers which may have different implementations on other targets. You may call

 $ cd test/all
 $ make run-avr # run for all AVRs
 $ make run-stm32 # run for all STM32, this may take a while
 $ python3 run_all.py stm32f4
  o  stm32f410t8y         75.9s
  o  stm32f410cbt         81.9s
  ...
  o  stm32f479zgt         19.7s
  o  stm32f479ngh         20.3s

SUCCESS 209
FAIL    0
------------------
Total:  209
 $ make run-failed # only run those that failed the last test

Target Compilation Logs

The individual compilation logs are all saved as artifacts, so that when something fails, it‘s easy to understand why. You can access them in the Artifacts tab of a build.

Continous Integration

modm uses CircleCI as a Continous Integration service. It compiles all the examples and executes the unit tests on Linux, then generates and compiles the full library for a blinky example for all targets that we support.

In the future we would like to execute the unit tests as well as the examples automatically on the target hardware as well. However, there isn‘t any third-party framework to make that easy, so we probably have to write our own.