How to test the CEA?
This document explains how to test the CEA during the development of the software. It is meant for developers interested in extending the software as well as a documentation of the process used for quality ensurance.
The tests are located in the cea/tests
folder and can be run with the cea test
command. Additionally, each time
a pull request is created or pushed to, the Jenkins (see below) runs the tests and reports the results back to GitHub.
The same is also done for each merge of a branch back to master.
Types of tests performed
The CEA employs two types of tests: Unit tests and an integration test. The boundaries between the two are not clearly defined in the CEA. This has historical reasons, mainly for the integration test.
The integration tests are defined in the file cea/tests/test_calc_thermal_loads.py
. They test the demand calculation
engine to make sure changes to the code don’t result in unexpected changes to the results. If results do change, the
program cea/tests/create_unittest_data.py
can be used to update the test data to compare against. The test data
is stored in the file cea/tests/test_calc_thermal_loads.config
. Note, you will need to verify the new results using
another method before merging back to master.
Strictly speaking, the integration tests in test_calc_thermal_loads.py
are coded as unit tests.
The rest of the python files in cea/tests
that start with test_
contain unit tests. See the python documentation
on the unittest module and also take a look at the pymotw page on the unittest
module.
There is a separate file called cea/tests/dodo.py
that deserves mentioning. It is an input file for the pydoit
automation tool. This script runs the unit tests and also runs a list of CEA tools on the example reference case as well
as a list of reference cases in a private GitHub repository. When you run the cea test
command, this is the script
that actually gets run. See below (chapter “dodo”) for more information.
Running tests locally
Before commit your work on a branch you should run the test suite locally. There are two ways to do this: Either using the command line interface (CLI) or using the unit test functionality in PyCharm itself.
To run the tests from the command line interface, use the command cea test
. This will run the unit tests as well
as the integration test with the reference case that ships with the CEA. To test the other reference cases, run the
command like this: cea test --reference-case all
- note, that you will need to set up authentication to the private
repository to do this. First, Generate a personal access token on GitHub, then create a text file in your %USERPROFILE% (home for POSIX systems) folder called cea_github.auth with two lines: The first is your GitHub user name, the second contains the personal access token you just generated.
Running the unit tests from PyCharm is as simple as right-clicking the folder cea/tests
in the project tool window
and choosing “Run Unittests in tests” from the context menu. Note that just running the unit tests will not also check
the other scripts.
The Jenkins
The Jenkins instance at http://ceajenkins.localtunnel.me/ is set up to run the test suite for every push to a pull request and every merge to master. Access to this website can be provided by the project administrators.
Since updates to pull requests can happen frequently, only the short version of the tests is run - the equivalent of
running cea test
locally. If the pull request page on GitHub states that “All checks have failed”, checking out the
branch and running cea test
will help with debugging the problem.
When a branch is merged into master, a full test is performed, equivalent to running cea test --reference-case all
.
This ensures that the master branch is kept in a defined state. This doesn’t mean there are no bugs. Just that those
we have found don’t creep back.
Dodo
The Jenkins is set up to run the script cea/tests/dodo.py
. This script uses a library called pydoit - an
implementation of a build system (e.g. make or ant) in python using python syntax to descript the targets and actions.
The dodo script is equivalent to a Makefile
in the standard make. We use this script to describe the integration
tests - a selection of scripts from the CEA are run on a set of reference cases to ensure that they at least run through
without producing errors. In addition, the set of unit tests is also run, actually testing the output of parts of the
functionality.
Test Driven Development
New code added to the CEA should be accompanied with unit tests. There are many benefits of using unit tests as a starting point for writing new code, a concept known as Test Driven Development (TDD). Developing in this style can turn the workflow “inside out”, starting bottom-up, testing at the level of equations and only aggregating later on in the development life-cycle. This can help reduce bugs and think about code internals. It also is a good way to run small “bits” of code quickly. Try it, and see how the style fits your work!