test_nb = read_nb('../../tests/image.ipynb')
assert test_nb.cells[0].outputs[0].data['image/png'][-1] == "\n" # Make sure it was not converted by acccident
clean_nb(test_nb)
assert test_nb.cells[0].outputs[0].data['image/png'][-1] != "\n"clean
Strip superfluous metadata from notebooks
To avoid pointless conflicts while working with jupyter notebooks (with different execution counts or cell metadata), it is recommended to clean the notebooks before committing anything (done automatically if you install the git hooks with nbdev-install-hooks). The following functions are used to do that. Cleaning also adds cell ids if missing (required by nbformat 4.5+).
Trust
nbdev_trust
def nbdev_trust(
fname:str=None, # A notebook name or glob to trust
force_all:bool=False, # Also trust notebooks that haven't changed
):Trust notebooks matching fname.
Clean
clean_nb
def clean_nb(
nb, # The notebook to clean
clear_all:bool=False, # Remove all cell metadata and cell outputs?
allowed_metadata_keys:list=None, # Preserve the list of keys in the main notebook metadata
allowed_cell_metadata_keys:list=None, # Preserve the list of keys in cell level metadata
clean_ids:bool=True, # Remove ids from plaintext reprs?
allowed_out_metadata_keys:list=None, # Preserve the list of keys in output metadata
repair:bool=True, # Fix structural problems first (see `repair_nb`)?
):Clean nb from superfluous metadata
Jupyter adds a trailing to images in cell outputs. Vscode-jupyter does not.
Notebooks should be brought to a common style to avoid unnecessary diffs:
The test notebook has metadata in both the main metadata section and contains cell level metadata in the second cell:
test_nb = read_nb('../../tests/metadata.ipynb')
assert {'meta', 'jekyll', 'nbdev', 'my_extra_key', 'my_removed_key'} <= test_nb.metadata.keys()
assert {'meta', 'hide_input', 'my_extra_cell_key', 'nbdev', 'my_removed_cell_key'} == test_nb.cells[1].metadata.keys()After cleaning the notebook, all extra metadata is removed, only some keys are allowed by default:
clean_nb(test_nb)
assert {'jekyll', 'kernelspec', 'nbdev'} == test_nb.metadata.keys()
assert {'hide_input', 'nbdev'} == test_nb.cells[1].metadata.keys()clean_nb also repairs structural problems by default (via repair_nb), so notebooks that Jupyter would reject, such as a markdown cell carrying an outputs attr, are fixed on every clean:
_nb = dict2nb(dict(cells=[dict(cell_type='markdown', source='hi', outputs=[], execution_count=1, id='m1', metadata={})],
metadata=dict(kernelspec=dict(name='python3', display_name='Python 3')), nbformat=4, nbformat_minor=5))
clean_nb(_nb)
assert 'outputs' not in _nb.cells[0] and 'execution_count' not in _nb.cells[0]
validate_nb(_nb)We can preserve some additional keys at the notebook or cell levels:
test_nb = read_nb('../../tests/metadata.ipynb')
clean_nb(test_nb, allowed_metadata_keys={'my_extra_key'}, allowed_cell_metadata_keys={'my_extra_cell_key'})
assert {'jekyll', 'kernelspec', 'nbdev', 'my_extra_key'} == test_nb.metadata.keys()
assert {'hide_input', 'nbdev', 'my_extra_cell_key'} == test_nb.cells[1].metadata.keys()Passing clear_all=True removes everything from the cell metadata:
test_nb = read_nb('../../tests/metadata.ipynb')
clean_nb(test_nb, clear_all=True)
assert {'jekyll', 'kernelspec', 'nbdev'} == test_nb.metadata.keys()
test_eq(test_nb.cells[1].metadata, {})Passing clean_ids=True removes ids from plaintext repr outputs, to avoid notebooks whose contents change on each run since they often lead to git merge conflicts. For example:
<PIL.PngImagePlugin.PngImageFile image mode=L size=28x28 at 0x7FB4F8979690>becomes:
<PIL.PngImagePlugin.PngImageFile image mode=L size=28x28>Cell IDs, on the other hand, are always added if missing
test_cell = {'source': 'x=1', 'cell_type': 'code', 'metadata': {}}
_clean_cell(test_cell, False, set(), True, set())
test_cell['id']'64e20756'process_write
def process_write(
warn_msg, proc_nb, f_in, f_out:NoneType=None, disp:bool=False
):Call self as a function.
Directive migrations
Deliberate, opt-in rewrites for moving to the directive conventions introduced in v3.3: directives can live in cell metadata (an nbdev dict key), notebook-scope directives like default_exp can live in notebook metadata, and comment spellings have one canonical form. None of these run by default; they’re flags on nbdev-clean for one-off migration runs, so the git hook never triggers them.
_to_meta and _to_comments move named directives between a cell’s comments and its nbdev metadata key, preserving values through the same mapping the parser uses (bare directives become true, and so on):
c = mk_cell('#| hide\n#| eval: false\n#| export: utils\n1+1')
_to_meta(c, ['hide','eval'])
test_eq(c.metadata['nbdev'], dict(hide='true', eval='false'))
test_eq(c.source, '#| export: utils\n1+1')
_to_comments(c, ['hide','eval'])
assert 'nbdev' not in c.metadata
test_eq(c.directives, {'export':'utils', 'hide':'', 'eval':'false'})_canon_dirs respells each directive line canonically without touching anything else, and _hoist_nb_meta moves default_exp to notebook metadata, deleting a first cell that held nothing but directives (a bare #| hide on an otherwise-empty cell hides nothing). _dir_moves bundles the migrations for the CLI, converting the raw loaded dict to NbCells first:
c = mk_cell('#| default_exp core\n#| eval:false\n1+1')
_canon_dirs(c)
test_eq(c.source, '#| default_exp: core\n#| eval: false\n1+1')
_nb = dict2nb(dict(cells=[mk_cell('#| hide\n#| default_exp: core'), mk_cell('#| export\n1+1')],
metadata={}, nbformat=4, nbformat_minor=5))
_hoist_nb_meta(_nb)
test_eq(_nb.metadata['nbdev'], dict(default_exp='core'))
test_eq(len(_nb.cells), 1)
test_eq(_nb.cells[0].source, '#| export\n1+1')nbdev_clean
def nbdev_clean(
fname:str=None, # A notebook name or glob to clean
clear_all:bool=False, # Remove all cell metadata and cell outputs?
disp:bool=False, # Print the cleaned outputs
stdin:bool=False, # Read notebook from input stream
repair:bool_arg=True, # Fix structural problems, e.g. stray outputs on non-code cells (see `repair_nb`)?
dirs:bool=False, # Rewrite comment directives in canonical form?
to_meta:str=None, # Space-separated directive names to move from comments to cell metadata
to_comments:str=None, # Space-separated directive names to move from cell metadata to comments
nb_meta:bool=False, # Move `default_exp` into notebook metadata?
):Clean all notebooks in fname to avoid merge conflicts
By default (fname left to None), all the notebooks in config.nbs_path are cleaned. You can opt in to fully clean the notebook by removing every bit of metadata and the cell outputs by passing clear_all=True.
If you want to keep some keys in the main notebook metadata you can set allowed_metadata_keys in [tool.nbdev] in pyproject.toml. Similarly for cell level metadata use allowed_cell_metadata_keys, and for output metadata use allowed_out_metadata_keys. For example, to preserve both k1 and k2 at both the notebook and cell level add the following to pyproject.toml:
[tool.nbdev]
allowed_metadata_keys = ["k1", "k2"]
allowed_cell_metadata_keys = ["k1", "k2"]
allowed_out_metadata_keys = ["k1", "k2"]clean_jupyter
def clean_jupyter(
path, model, **kwargs
):Clean Jupyter model pre save to path
This cleans notebooks on-save to avoid unnecessary merge conflicts. The easiest way to install it for both Jupyter Notebook and Lab is by running nbdev-install-hooks. It works by implementing a pre_save_hook from Jupyter’s file save hook API.
Hooks
nbdev_install_hooks
def nbdev_install_hooks(
merge:bool_arg=True, # Install the notebook merge driver?
diff:bool_arg=True, # Install the notebook diff driver?
globally:bool_arg=False, # Define the drivers in `~/.gitconfig` and the global attributes file, instead of repo files?
):Install Jupyter and git hooks to automatically clean, trust, and fix merge conflicts in notebooks
See clean_jupyter and nbdev-merge for more about how each hook works.
Both git drivers are registered under the name jupyternotebook, the same name nbdime uses. Repo installs define them in a repo-local .gitconfig (wired in via include.path) and activate them in the committed .gitattributes; --globally instead defines them in ~/.gitconfig and activates them in the global attributes file (core.attributesFile, defaulting to ~/.config/git/attributes). Since the attribute lines just name a driver, a committed .gitattributes means “use your preferred notebook driver”: whichever tool defined jupyternotebook last in a config git consults wins, so switching between nbdev and nbdime is just re-running either tool’s enable command. Global installs skip the repo-only post-merge trust hook.