test

Run unit tests on notebooks in parallel

When a test run hangs, ctrl-c should say what each notebook was executing. Each worker installs a SIGINT handler that prints the in-flight notebook’s stack — the sync frames plus every pending asyncio task, since a hang inside an awaited coroutine lives on the task, not the sync stack — as a single block (one os.write, so parallel workers’ dumps don’t interleave) and then exits immediately, rather than letting the interrupt surface inside the cell where the shell would catch it and carry on. An idle worker exits silently.

---

source

test_nb

def test_nb(
    fn, # file name of notebook to test
    skip_flags:NoneType=None, # list of flags marking cells to skip
    force_flags:NoneType=None, # list of flags marking cells to always run
    do_print:bool=False, # print completion?
    showerr:bool=True, # print errors to stderr?
    basepath:NoneType=None, # path to add to sys.path
    verbose:bool=False, # stream stdout/stderr from cells to console?
    save:bool=False, # write outputs back to notebook on success?
    profile:bool=None, # load the IPython profile, as `ipykernel` does? (default: `exec_profile` config key)
    cell_timeout:int=600, # seconds before each cell times out (None: no limit)
):

Execute tests in notebook in fn except those with skip_flags

test_nb is sync: every cell runs on the CaptureShell’s own loop thread, and the caller simply blocks on each result. parallel workers call it directly, and because the worker’s main thread stays free while cells run, the SIGINT handler above can dump the shell thread’s stack and its pending tasks in the middle of a hang. cell_timeout bounds each cell even when it blocks the loop in sync code – the timed-out notebook fails with a TimeoutError naming the cell and the stuck tasks, instead of hanging the run.

test_nb can test a notebook, and skip over certain flags. A notebook whose frontmatter sets skip_exec: true (e.g. as a - skip_exec: true list item in its title cell) is skipped entirely and reported as passing; use it for notebooks that can’t run under test at all, such as those needing credentials or live services:

_nb = Path('../../tests/directives.ipynb')
success,duration = test_nb(_nb, skip_flags=['notest'])
assert success

In that notebook the cell flagged notest raises an exception, which will be returned as a bool:

_nb = Path('../../tests/directives.ipynb')
success,duration = test_nb(_nb, showerr=False)
assert not success

import tempfile
from fastcore.xtras import modified_env

test_nb loads the IPython profile by default, like ipykernel does, so notebooks are tested the way their author’s kernel ran them (startup files, extensions, shell config). Set exec_profile = false under [tool.nbdev], or pass profile=False, to run without it:

with tempfile.TemporaryDirectory() as td:
    td = Path(td)
    (td/'profile_default'/'startup').mkdir(parents=True)
    (td/'profile_default'/'startup'/'00.py').write_text('prof_x = 7\n')
    nbp = td/'prof.ipynb'
    write_nb(new_nb([mk_cell('assert prof_x==7')]), nbp)
    with modified_env(IPYTHONDIR=str(td)):
        assert     (test_nb(nbp, showerr=False))[0]
        assert not (test_nb(nbp, showerr=False, profile=False))[0]

Sometimes you may wish to override one or more of the skip_flags, in which case you can use the argument force_flags which will remove the appropriate tag(s) from skip_flags. This is useful because skip_flags are meant to be set in the tst_flags field of [tool.nbdev] in pyproject.toml, whereas force_flags are usually passed in by the user.

---

source

nbdev_test

def nbdev_test(
    path:str=None, # A notebook name or glob to test
    flags:str='', # Space separated list of test flags to run that are normally ignored
    n_workers:int=None, # Number of workers
    timing:bool=False, # Time each notebook to see which are slow
    do_print:bool=False, # Print start and end of each notebook
    pause:float=0.01, # Pause time (in seconds) between notebooks to avoid race conditions
    ignore_fname:str='.notest', # Filename that will result in siblings being ignored
    verbose:bool=False, # Print stdout/stderr from notebook cells?
    save:bool=False, # Write outputs back to notebooks on success?
    cell_timeout:int=600, # Seconds before each cell times out (0: no limit)
    symlinks:bool=False, # Follow symlinks?
    file_glob:str='*.ipynb', # Only include files matching glob
    file_re:str=None, # Only include files matching regex
    folder_re:str=None, # Only enter folders matching regex
    skip_file_glob:str=None, # Skip files matching glob
    skip_file_re:str='^[_.]', # Skip files matching regex
    skip_folder_re:str='^[_.]', # Skip folders matching regex
):

Test in parallel notebooks matching path, passing along flags

nbdev_test(n_workers=0)

Success.

You can even run nbdev-test in non nbdev projects, for example, you can test an individual notebook like so:

nbdev-test --path ../../tests/minimal.ipynb --do_print

Or you can test an entire directory of notebooks filtered for only those that match a regular expression:

nbdev-test --path ../../tests --file_re '.*test.ipynb' --do_print

Eval

test_nb decides which cells run through the eval cascade (fastcore.nbio.does_cell_eval): a cell’s own #| eval: directive wins; otherwise the notebook-level eval directive — in frontmatter, or the notebook’s metadata.nbdev mapping — sets the default; with neither, cells run. #| eval: false therefore skips one cell, as it always has, while a notebook-level eval: false flips the whole notebook to opt-in: only cells marked #| eval: true run, which suits a slow or service-dependent notebook where just a few cells are worth testing. Unlike skip_exec: true, which skips a notebook unconditionally, marked cells still run — here the unmarked cell would raise if executed, so the passing test is the proof it was skipped:

with tempfile.TemporaryDirectory() as td:
    cells = [mk_cell('---\neval: false\n---', 'raw'), mk_cell('raise Exception("unmarked: must not run")'),
        mk_cell('#| eval: true\nx = 1')]
    fn = Path(td)/'optin.ipynb'
    write_nb(new_nb(cells), fn)
    success,_ = test_nb(fn)
assert success