# clean To avoid pointless conflicts while working with jupyter notebooks (with different execution counts or cell metadata), it is recommended to clean the notebooks before committing anything (done automatically if you install the git hooks with `nbdev-install-hooks`). The following functions are used to do that. Cleaning also adds cell `id`s if missing (required by nbformat 4.5+). ## Trust ------------------------------------------------------------------------ source ### nbdev_trust ``` python def nbdev_trust( fname:str=None, # A notebook name or glob to trust force_all:bool=False, # Also trust notebooks that haven't changed ): ``` *Trust notebooks matching `fname`.* ## Clean ------------------------------------------------------------------------ source ### clean_nb ``` python def clean_nb( nb, # The notebook to clean clear_all:bool=False, # Remove all cell metadata and cell outputs? allowed_metadata_keys:list=None, # Preserve the list of keys in the main notebook metadata allowed_cell_metadata_keys:list=None, # Preserve the list of keys in cell level metadata clean_ids:bool=True, # Remove ids from plaintext reprs? allowed_out_metadata_keys:list=None, # Preserve the list of keys in output metadata repair:bool=True, # Fix structural problems first (see `repair_nb`)? ): ``` *Clean `nb` from superfluous metadata* Jupyter adds a trailing to images in cell outputs. Vscode-jupyter does not.\ Notebooks should be brought to a common style to avoid unnecessary diffs: ``` python test_nb = read_nb('../../tests/image.ipynb') assert test_nb.cells[0].outputs[0].data['image/png'][-1] == "\n" # Make sure it was not converted by acccident clean_nb(test_nb) assert test_nb.cells[0].outputs[0].data['image/png'][-1] != "\n" ``` The test notebook has metadata in both the main metadata section and contains cell level metadata in the second cell: ``` python test_nb = read_nb('../../tests/metadata.ipynb') assert {'meta', 'jekyll', 'nbdev', 'my_extra_key', 'my_removed_key'} <= test_nb.metadata.keys() assert {'meta', 'hide_input', 'my_extra_cell_key', 'nbdev', 'my_removed_cell_key'} == test_nb.cells[1].metadata.keys() ``` After cleaning the notebook, all extra metadata is removed, only some keys are allowed by default: ``` python clean_nb(test_nb) assert {'jekyll', 'kernelspec', 'nbdev'} == test_nb.metadata.keys() assert {'hide_input', 'nbdev'} == test_nb.cells[1].metadata.keys() ``` [`clean_nb`](https://nbdev.fast.ai/api/clean.html#clean_nb) also repairs structural problems by default (via `repair_nb`), so notebooks that Jupyter would reject, such as a markdown cell carrying an `outputs` attr, are fixed on every clean: ``` python _nb = dict2nb(dict(cells=[dict(cell_type='markdown', source='hi', outputs=[], execution_count=1, id='m1', metadata={})], metadata=dict(kernelspec=dict(name='python3', display_name='Python 3')), nbformat=4, nbformat_minor=5)) clean_nb(_nb) assert 'outputs' not in _nb.cells[0] and 'execution_count' not in _nb.cells[0] validate_nb(_nb) ``` We can preserve some additional keys at the notebook or cell levels: ``` python test_nb = read_nb('../../tests/metadata.ipynb') clean_nb(test_nb, allowed_metadata_keys={'my_extra_key'}, allowed_cell_metadata_keys={'my_extra_cell_key'}) assert {'jekyll', 'kernelspec', 'nbdev', 'my_extra_key'} == test_nb.metadata.keys() assert {'hide_input', 'nbdev', 'my_extra_cell_key'} == test_nb.cells[1].metadata.keys() ``` Passing `clear_all=True` removes everything from the cell metadata: ``` python test_nb = read_nb('../../tests/metadata.ipynb') clean_nb(test_nb, clear_all=True) assert {'jekyll', 'kernelspec', 'nbdev'} == test_nb.metadata.keys() test_eq(test_nb.cells[1].metadata, {}) ``` Passing `clean_ids=True` removes `id`s from plaintext repr outputs, to avoid notebooks whose contents change on each run since they often lead to git merge conflicts. For example: becomes: *Cell* IDs, on the other hand, are always added if missing ``` python test_cell = {'source': 'x=1', 'cell_type': 'code', 'metadata': {}} _clean_cell(test_cell, False, set(), True, set()) test_cell['id'] ``` '64e20756' ------------------------------------------------------------------------ source ### process_write ``` python def process_write( warn_msg, proc_nb, f_in, f_out:NoneType=None, disp:bool=False ): ``` ## Directive migrations Deliberate, opt-in rewrites for moving to the directive conventions introduced in v3.3: directives can live in cell metadata (an `nbdev` dict key), notebook-scope directives like `default_exp` can live in notebook metadata, and comment spellings have one canonical form. None of these run by default; they’re flags on `nbdev-clean` for one-off migration runs, so the git hook never triggers them. [`_to_meta`](https://nbdev.fast.ai/api/clean.html#_to_meta) and [`_to_comments`](https://nbdev.fast.ai/api/clean.html#_to_comments) move named directives between a cell’s comments and its `nbdev` metadata key, preserving values through the same mapping the parser uses (bare directives become `true`, and so on): ``` python c = mk_cell('#| hide\n#| eval: false\n#| export: utils\n1+1') _to_meta(c, ['hide','eval']) test_eq(c.metadata['nbdev'], dict(hide='true', eval='false')) test_eq(c.source, '#| export: utils\n1+1') _to_comments(c, ['hide','eval']) assert 'nbdev' not in c.metadata test_eq(c.directives, {'export':'utils', 'hide':'', 'eval':'false'}) ``` [`_canon_dirs`](https://nbdev.fast.ai/api/clean.html#_canon_dirs) respells each directive line canonically without touching anything else, and [`_hoist_nb_meta`](https://nbdev.fast.ai/api/clean.html#_hoist_nb_meta) moves `default_exp` to notebook metadata, deleting a first cell that held nothing but directives (a bare `#| hide` on an otherwise-empty cell hides nothing). [`_dir_moves`](https://nbdev.fast.ai/api/clean.html#_dir_moves) bundles the migrations for the CLI, converting the raw loaded dict to `NbCell`s first: ``` python c = mk_cell('#| default_exp core\n#| eval:false\n1+1') _canon_dirs(c) test_eq(c.source, '#| default_exp: core\n#| eval: false\n1+1') _nb = dict2nb(dict(cells=[mk_cell('#| hide\n#| default_exp: core'), mk_cell('#| export\n1+1')], metadata={}, nbformat=4, nbformat_minor=5)) _hoist_nb_meta(_nb) test_eq(_nb.metadata['nbdev'], dict(default_exp='core')) test_eq(len(_nb.cells), 1) test_eq(_nb.cells[0].source, '#| export\n1+1') ``` ------------------------------------------------------------------------ source ### nbdev_clean ``` python def nbdev_clean( fname:str=None, # A notebook name or glob to clean clear_all:bool=False, # Remove all cell metadata and cell outputs? disp:bool=False, # Print the cleaned outputs stdin:bool=False, # Read notebook from input stream repair:=True, # Fix structural problems, e.g. stray outputs on non-code cells (see `repair_nb`)? dirs:bool=False, # Rewrite comment directives in canonical form? to_meta:str=None, # Space-separated directive names to move from comments to cell metadata to_comments:str=None, # Space-separated directive names to move from cell metadata to comments nb_meta:bool=False, # Move `default_exp` into notebook metadata? ): ``` *Clean all notebooks in `fname` to avoid merge conflicts* By default (`fname` left to `None`), all the notebooks in `config.nbs_path` are cleaned. You can opt in to fully clean the notebook by removing every bit of metadata and the cell outputs by passing `clear_all=True`. If you want to keep some keys in the main notebook metadata you can set `allowed_metadata_keys` in `[tool.nbdev]` in `pyproject.toml`. Similarly for cell level metadata use `allowed_cell_metadata_keys`, and for output metadata use `allowed_out_metadata_keys`. For example, to preserve both `k1` and `k2` at both the notebook and cell level add the following to `pyproject.toml`: ``` toml [tool.nbdev] allowed_metadata_keys = ["k1", "k2"] allowed_cell_metadata_keys = ["k1", "k2"] allowed_out_metadata_keys = ["k1", "k2"] ``` ------------------------------------------------------------------------ source ### clean_jupyter ``` python def clean_jupyter( path, model, **kwargs ): ``` *Clean Jupyter `model` pre save to `path`* This cleans notebooks on-save to avoid unnecessary merge conflicts. The easiest way to install it for both Jupyter Notebook and Lab is by running `nbdev-install-hooks`. It works by implementing a `pre_save_hook` from Jupyter’s [file save hook API](https://jupyter-server.readthedocs.io/en/latest/developers/savehooks.html). ## Hooks ------------------------------------------------------------------------ source ### nbdev_install_hooks ``` python def nbdev_install_hooks( merge:=True, # Install the notebook merge driver? diff:=True, # Install the notebook diff driver? globally:=False, # Define the drivers in `~/.gitconfig` and the global attributes file, instead of repo files? ): ``` *Install Jupyter and git hooks to automatically clean, trust, and fix merge conflicts in notebooks* See [`clean_jupyter`](https://nbdev.fast.ai/api/clean.html#clean_jupyter) and `nbdev-merge` for more about how each hook works. Both git drivers are registered under the name `jupyternotebook`, the same name nbdime uses. Repo installs define them in a repo-local `.gitconfig` (wired in via `include.path`) and activate them in the committed `.gitattributes`; `--globally` instead defines them in `~/.gitconfig` and activates them in the global attributes file (`core.attributesFile`, defaulting to `~/.config/git/attributes`). Since the attribute lines just name a driver, a committed `.gitattributes` means “use your preferred notebook driver”: whichever tool defined `jupyternotebook` last in a config git consults wins, so switching between nbdev and nbdime is just re-running either tool’s enable command. Global installs skip the repo-only `post-merge` trust hook.