Config File And Using SciUnit In A Shell¶

Create Config File And Execute Tests In A Shell¶

We can build a scientific computing project with a SciUnit config file. Then, we will be able to run sciunit In A Shell

Here is an example of well written SciUnit config file. This file was generated by executing sciunit create in the shell. A SciUnit config file is always named sciunit.ini.

[misc]
config-version = 1.0
nb-name = scidash

[root]
path = .

[models]
module = models

[tests]
module = tests

[suites]
module = suites

config-version is the version of the config file.

nb-name is the name of the IPython Notebook file that can be create with sciunit make-nb.

root is the root of the project. The path is the path to the project from the directory that contains this config file.

module in the models section is the path from the root of the project to the file that contains models, which is a list of Model instances.

module in the tests section is the path the root of the project to the file that contains tests, which is a list of Test instances.

module in the suites section is the path the root of the project to the file that contains suites, which is a list of TestSuite instances.

Let’s use the config file above and create corresponding files that contain definitions models, tests, and suites.

In the root directory of the project, let’s create three files.

tests.py

import sciunit
from sciunit.scores import BooleanScore
from sciunit.capabilities import ProducesNumber



class EqualsTest(sciunit.Test):
    """Tests if the model predicts
    the same number as the observation."""

    required_capabilities = (ProducesNumber,)
    score_type = BooleanScore

    def generate_prediction(self, model):
        return model.produce_number() # The model has this method if it inherits from the 'ProducesNumber' capability.

    def compute_score(self, observation, prediction):
        score = self.score_type(observation['value'] == prediction)
        score.description = 'Passing score if the prediction equals the observation'
        return score

tests = []

suites.py

import sciunit
from tests import EqualsTest


equals_1_test = EqualsTest({'value':1}, name='=1')
equals_2_test = EqualsTest({'value':2}, name='=2')
equals_37_test = EqualsTest({'value':37}, name='=37')

equals_suite = sciunit.TestSuite([equals_1_test, equals_2_test, equals_37_test], name="Equals test suite")

suites = [equals_suite]

models.py

import sciunit
from sciunit.capabilities import ProducesNumber


class ConstModel(sciunit.Model,
                ProducesNumber):
    """A model that always produces a constant number as output."""

    def __init__(self, constant, name=None):
        self.constant = constant
        super(ConstModel, self).__init__(name=name, constant=constant)

    def produce_number(self):
        return self.constant

const_model_1 = ConstModel(1, name='Constant Model 1')
const_model_2 = ConstModel(2, name='Constant Model 2')
const_model_37 = ConstModel(37, name="Constant Model 37")

models = [const_model_1, const_model_2, const_model_37]

We have tests at the end of tests.py, models at the end of models.py, and suites at the end of suites.py. Since we are using test suites instead of tests, tests is an empty list in this example. They will be taken by sciunit when command sciunit run is being executing

Execute sciunit run in the root directory, and then sciunit will run each test in the suites against each model and give us the result.

$ sciunit run


Executing test =1 on model Constant Model 1... Score is Pass
Executing test =2 on model Constant Model 1... Score is Fail
Executing test =37 on model Constant Model 1... Score is Fail
Executing test =1 on model Constant Model 2... Score is Fail
Executing test =2 on model Constant Model 2... Score is Pass
Executing test =37 on model Constant Model 2... Score is Fail
Executing test =1 on model Constant Model 37... Score is Fail
Executing test =2 on model Constant Model 37... Score is Fail
Executing test =37 on model Constant Model 37... Score is Pass

Suite Equals test suite:
                    =1    =2   =37
Constant Model 1   Pass  Fail  Fail
Constant Model 2   Fail  Pass  Fail
Constant Model 37  Fail  Fail  Pass

Create and Run IPython Notebook File¶

Next, let’s move to creating and executing IPython Notebook file with sciunit make-nb and sciunit run-nb commands.

Let’s add a file, __init__.py, to our project directory and import everything including suites, tests, and models in the file. This is necessary because the made notebook file will try to import everything in __init__.py and run each suite (a collection of tests instances) against each model.

__init__.py

from . import models
from . import tests
from . import suites

Now, let’s execute sciunit make-nb SciUnit will automatically generate a notebook file.

$ sciunit make-nb
Created Jupyter notebook at:
/the_path_to_the_project.../test_sciunit.ipynb

The notebook file will contains two blocks of code:

Note:

the name of generated notebook file will be the value of nb-name attribute in the config file, sciunit.ini
The path to the project’s root can be different on different machine. So, The notebook file generated usually only be valid on the machine where it is generated. If you want to execute it on different machine, try to re-generate it or change the path.

Let’s execute sciunit run-nb command.

$ sciunit run-nb
Entering run function
/the_path_to_the_project_config_file..././test_sciunit.ipynb
/the_path_to_the_project_config_file.../.

The result of running the notebook will be in the notebook file. You can open it by many tools like VS Code and Jupyter Lab