test

Run unit tests on notebooks in parallel

Press Ctrl-C during a hung test run to see what each notebook is executing. Each worker prints the synchronous stack and the await chains of pending asyncio tasks. An awaited coroutine can be stuck somewhere the synchronous stack doesn’t show.

Workers print their diagnostics in separate blocks and exit immediately. The shell must not catch the interrupt and continue running cells. Idle workers exit silently.

---

source

nb_test_cells

def nb_test_cells(
    fn, # Notebook path
    skip_flags:NoneType=None, # Flags marking cells to skip
    force_flags:NoneType=None, # Flags marking cells to always run
):

The cells test_nb would run in the notebook at fn

nb_test_cells lists the cells selected for testing. It excludes non-Python notebooks and notebooks with skip_exec: true in frontmatter. Within a Python notebook, eval directives and skip flags determine which code cells run.

Pass skip_flags=['notest'] to leave out cells marked notest:

_nb = Path('../../tests/directives.ipynb')
cells = nb_test_cells(_nb, skip_flags=['notest'])
assert not any('notest' in c.source for c in cells)
len(cells), len(nb_test_cells(_nb))

---

source

test_nb

def test_nb(
    fn, # file name of notebook to test
    skip_flags:NoneType=None, # list of flags marking cells to skip
    force_flags:NoneType=None, # list of flags marking cells to always run
    do_print:bool=False, # print completion?
    showerr:bool=True, # print errors to stderr?
    basepath:NoneType=None, # path to add to sys.path
    verbose:bool=False, # stream stdout/stderr from cells to console?
    save:bool=False, # write outputs back to notebook on success?
    profile:bool=None, # load the IPython profile, as `ipykernel` does? (default: `exec_profile` config key)
    cell_timeout:int=600, # seconds before each cell times out (None: no limit)
    cell_timing_min:float=None, # print cells slower than this many seconds (None: no timing output)
):

Execute tests in notebook in fn except those with skip_flags

test_nb runs cells on CaptureShell’s loop thread and waits for each result. The worker’s main thread can still handle Ctrl-C while a cell is stuck.

cell_timeout limits execution even when synchronous code blocks the loop. A timeout fails the notebook with a TimeoutError identifying the cell and stuck tasks.

Failure reports use sys.__stderr__ to bypass cell output capture. Uncancellable cell code can still be running after a timeout, with its captured sys.stderr still installed.

test_nb returns a success flag and duration. A notebook with skip_exec: true skips all execution and reports success. Use this for notebooks that need unavailable credentials or live services.

This test succeeds when we skip the notest cell:

_nb = Path('../../tests/directives.ipynb')
success,duration = test_nb(_nb, skip_flags=['notest'])
assert success

The notest cell raises an exception. Including it makes the test fail:

_nb = Path('../../tests/directives.ipynb')
success,duration = test_nb(_nb, showerr=False)
assert not success

import tempfile
from fastcore.xtras import modified_env

Use cell_timing_min to report slow cells. Reports give the notebook name, cell ID and elapsed seconds. Try --cell-timing-min=0.1 to start.

The nbdev-test command reads defaults for cell_timeout and cell_timing_min from configuration. Set them in ~/.config/nbdev/config.toml or your project’s [tool.nbdev] table.

test_nb loads the IPython profile by default, including startup files, extensions and shell configuration. This matches ipykernel behavior. To run without the profile, set exec_profile = false under [tool.nbdev] or pass profile=False:

with tempfile.TemporaryDirectory() as td:
    td = Path(td)
    (td/'profile_default'/'startup').mkdir(parents=True)
    (td/'profile_default'/'startup'/'00.py').write_text('prof_x = 7\n')
    nbp = td/'prof.ipynb'
    write_nb(new_nb([mk_cell('assert prof_x==7')]), nbp)
    with modified_env(IPYTHONDIR=str(td)):
        assert     (test_nb(nbp, showerr=False))[0]
        assert not (test_nb(nbp, showerr=False, profile=False))[0]

Set default skip flags in tst_flags under [tool.nbdev] in pyproject.toml. To run cells with those flags, pass them in force_flags to test_nb, or use --flags with nbdev-test.

---

source

nbdev_test

def nbdev_test(
    path:str=None, # A notebook name or glob to test
    flags:str='', # Space separated list of test flags to run that are normally ignored
    n_workers:int=None, # Number of workers
    timing:bool=False, # Time each notebook to see which are slow
    do_print:bool=False, # Print start and end of each notebook
    pause:float=0.01, # Pause time (in seconds) between notebooks to avoid race conditions
    ignore_fname:str='.notest', # Filename that will result in siblings being ignored
    verbose:bool=False, # Print stdout/stderr from notebook cells?
    save:bool=False, # Write outputs back to notebooks on success?
    cell_timeout:int=None, # Seconds before each cell times out (0: no limit; default `cell_timeout` config, 600)
    cell_timing_min:float=None, # Print cells slower than this many seconds (default `cell_timing_min` config, else none)
    *, symlinks:bool=False, # Follow symlinks?
    file_glob:str='*.ipynb', # Only include files matching glob
    file_re:str=None, # Only include files matching regex
    folder_re:str=None, # Only enter folders matching regex
    skip_file_glob:str=None, # Skip files matching glob
    skip_file_re:str='^[_.]', # Skip files matching regex
    skip_folder_re:str='^[_.]', # Skip folders matching regex
):

Test in parallel notebooks matching path, passing along flags

Use test_setup for preparation shared by several notebooks, such as downloading a dataset. Running this before workers start avoids competing downloads that can exhaust cell timeouts.

Set test_setup under [tool.nbdev] to a module:callable. nbdev-test calls it once in the parent process with all cells selected by nb_test_cells. Its output goes directly to the console, where it can report progress.

nbdev_test(n_workers=0)

Success.

You can even run nbdev-test in non nbdev projects, for example, you can test an individual notebook like so:

nbdev-test ../../tests/minimal.ipynb --do-print

Or you can test an entire directory of notebooks filtered for only those that match a regular expression:

nbdev-test ../../tests --file-re '.*test.ipynb' --do-print

Eval

A cell’s eval directive overrides the notebook default. Set that default in frontmatter or metadata.nbdev. Without either, cells run. test_nb uses fastcore.nbio.does_cell_eval for this decision.

For a slow or service-dependent notebook, set notebook-level eval: false and mark individual testable cells with #| eval: true. This differs from skip_exec: true, which skips the entire notebook regardless of cell directives.

Here the marked cell runs. The unmarked cell must not run:

with tempfile.TemporaryDirectory() as td:
    cells = [mk_cell('---\neval: false\n---', 'raw'), mk_cell('raise Exception("unmarked: must not run")'),
        mk_cell('#| eval: true\nx = 1')]
    fn = Path(td)/'optin.ipynb'
    write_nb(new_nb(cells), fn)
    success,_ = test_nb(fn)
assert success