The functions that transform notebooks in a library

The most important function defined in this module is notebooks2script, so you may want to jump to it before scrolling though the rest, which explain the details behind the scenes of the conversion from notebooks to library. The main things to remember are:

  • put an # export or %nbdev_export flag on each cell you want exported
  • put an # exports or %nbdev_export_and_show flag on each cell you want exported with the source code shown in the docs
  • put an # exporti or %nbdev_export_internal flag on each cell you want exported without it being added to __all__, and without it showing up in the docs.
  • one cell should contain a # default_exp or %nbdev_default_export flag followed by the name of the module (with points for submodules and without the py extension) everything should be exported in (if one specific cell needs to be exported in a different module, just indicate it after the %nbdev_export flag: %nbdev_export special.module)
  • all left members of an equality, functions and classes will be exported and variables that are not private will be put in the __all__ automatically
  • to add something to __all__ if it's not picked automatically, write an exported cell with something like %nbdev_add2all "my_name"

Basic foundations

For bootstrapping nbdev we have a few basic foundations defined in imports, which we test a show here. First, a simple config file class, Config that read the content of your settings.ini file and make it accessible:



Store the basic information for nbdev to work

create_config("github", "nbdev", user='fastai', path='..', tst_flags='tst', cfg_name='test_settings.ini')
cfg = Config(cfg_name='test_settings.ini')
test_eq(cfg.lib_name, 'nbdev')
test_eq(cfg.git_url, "")
test_eq(cfg.lib_path, Path.cwd().parent/'nbdev')
test_eq(cfg.nbs_path, Path.cwd())
test_eq(cfg.doc_path, Path.cwd().parent/'docs')
test_eq(cfg.custom_sidebar, 'False')

We derive some useful variables to check what environment we're in:

if not os.environ.get("IN_TEST", None):
    assert IN_NOTEBOOK
    assert not IN_COLAB
    assert IN_IPYTHON

Then we have a few util functions.


last_index(x, o)

Finds the last index of occurence of x in o (returns -1 if no occurence)

test_eq(last_index(1, [1,2,1,3,1,4]), 4)
test_eq(last_index(2, [1,2,1,3,1,4]), 1)
test_eq(last_index(5, [1,2,1,3,1,4]), -1)


compose(*funcs, order=None)

Create a function that composes all functions in funcs, passing along remaining *args and **kwargs to all

f1 = lambda o,p=0: (o*2)+p
f2 = lambda o,p=1: (o+1)/p
test_eq(f2(f1(3)), compose(f1,f2)(3))
test_eq(f2(f1(3,p=3),p=3), compose(f1,f2)(3,p=3))
test_eq(f2(f1(3,  3),  3), compose(f1,f2)(3,  3))


parallel(f, items, *args, n_workers=None, **kwargs)

Applies func in parallel to items, using n_workers

import time,random

def add_one(x, a=1): 
    return x+a

inp,exp = range(50),range(1,51)
test_eq(parallel(add_one, inp, n_workers=2), list(exp))
test_eq(parallel(add_one, inp, n_workers=0), list(exp))
test_eq(parallel(add_one, inp, n_workers=1, a=2), list(range(2,52)))
test_eq(parallel(add_one, inp, n_workers=0, a=2), list(range(2,52)))



First element of x, or None if missing

Reading a notebook

What's a notebook?

A jupyter notebook is a json file behind the scenes. We can just read it with the json module, which will return a nested dictionary of dictionaries/lists of dictionaries, but there are some small differences between reading the json and using the tools from nbformat so we'll use this one.



Read the notebook in fname.

fname can be a string or a pathlib object.

test_nb = read_nb('00_export.ipynb')

The root has four keys: cells contains the cells of the notebook, metadata some stuff around the version of python used to execute the notebook, nbformat and nbformat_minor the version of nbformat.

dict_keys(['cells', 'metadata', 'nbformat', 'nbformat_minor'])
{'jupytext': {'split_at_heading': True},
 'kernelspec': {'display_name': 'Python 3',
  'language': 'python',
  'name': 'python3'},
 'language_info': {'codemirror_mode': {'name': 'ipython', 'version': 3},
  'file_extension': '.py',
  'mimetype': 'text/x-python',
  'name': 'python',
  'nbconvert_exporter': 'python',
  'pygments_lexer': 'ipython3',
  'version': '3.7.7'},
 'toc': {'base_numbering': 1,
  'nav_menu': {},
  'number_sections': True,
  'sideBar': True,
  'skip_h1_title': False,
  'title_cell': 'Table of Contents',
  'title_sidebar': 'Contents',
  'toc_cell': False,
  'toc_position': {},
  'toc_section_display': True,
  'toc_window_display': False}}

The cells key then contains a list of cells. Each one is a new dictionary that contains entries like the type (code or markdown), the source (what is written in the cell) and the output (for code cells).

{'cell_type': 'code',
 'execution_count': 1,
 'metadata': {'hide_input': True},
 'outputs': [{'name': 'stdout',
   'output_type': 'stream',
   'text': 'Cells will be exported to nbdev.export,\nunless a different module is specified after an export flag: `%nbdev_export special.module`\n'}],
 'source': 'from nbdev import *\n%nbdev_default_export export\n%nbdev_default_class_level 3'}

Finding patterns

The following functions are used to catch the flags used in the code cells.


check_re(cell, pat, code_only=True)

Check if cell contains a line with regex pat

pat can be a string or a compiled regex, if code_only=True, this function ignores markdown cells.

cell = test_nb['cells'][1].copy()
assert check_re(cell, '%nbdev_export') is not None
assert check_re(cell, re.compile('%nbdev_export')) is not None
assert check_re(cell, '# bla') is None
cell['cell_type'] = 'markdown'
assert check_re(cell, '%nbdev_export') is None
assert check_re(cell, '%nbdev_export', code_only=False) is not None


check_re_multi(cell, pats, code_only=True)

Check if cell contains a line matching any regex in pats, returning the first match found

cell = test_nb['cells'][0].copy()
cell['source'] = "a b c"
assert check_re(cell, 'a') is not None
assert check_re(cell, 'd') is None
# show that searching with paterns ['d','b','a'] will match 'b'
# i.e. 'd' is not found and we don't search for 'a'
assert check_re_multi(cell, ['d','b','a']).span() == (2,3)

This function returns a regex object that can be used to find nbdev flags in multiline text

  • magic_flag pass True to find magic flags, False to find comment flags,
  • body regex fragment to match one or more flags,
  • n_params number of flag parameters to match and catch (-1 for any number of params),
  • comment explains what the compiled regex should do.


is_export(cell, default)

Check if cell is to be exported and returns the name of the module to export it if provided

is_export returns;

  • a tuple of ("module name", "external boolean") if cell is to be exported or
    • "external boolean" will be False for an internal export
  • None if cell will not be exported.

The cells to export are marked with an %nbdev_export, %nbdev_export_and_show or %nbdev_export_internal flag, potentially with a module name where we want it exported. The default module is given in a cell of the form %nbdev_default_export bla inside the notebook (usually at the top), though in this function, it needs the be passed (the final script will read the whole notebook to find it).

  • a cell marked with %nbdev_export, %nbdev_export_and_show or %nbdev_export_internal will be exported to the default module
  • a cell marked with %nbdev_export special.module, %nbdev_export_and_show special.module or %nbdev_export_internal special.module will be exported in special.module (located in lib_name/special/
  • a cell marked with %nbdev_export will have it's signature added to the documentation
  • a cell marked with %nbdev_export_and_show will additionally have it's source code added to the documentation
  • a cell marked with %nbdev_export_internal will not show up in the documentation, and will also not be added to __all__.
cell = test_nb['cells'][1].copy()
test_eq(is_export(cell, 'export'), ('export', True))
cell['source'] = "%nbdev_export\nfrom nbdev.imports import *"
test_eq(is_export(cell, 'export'), ('export', True))
cell['source'] = "# exports"
test_eq(is_export(cell, 'export'), ('export', True))
cell['source'] = "%nbdev_export_and_show"
test_eq(is_export(cell, 'export'), ('export', True))
cell['source'] = "# exporti"
test_eq(is_export(cell, 'export'), ('export', False))
cell['source'] = "%nbdev_export_internal"
test_eq(is_export(cell, 'export'), ('export', False))
cell['source'] = "# export mod"
test_eq(is_export(cell, 'export'), ('mod', True))
cell['source'] = "%nbdev_export mod\nfrom nbdev.imports import *"
test_eq(is_export(cell, 'export'), ('mod', True))
cell['source'] = "# export mod.file"
test_eq(is_export(cell, 'export'), (f'mod{os.path.sep}file', True))
cell['source'] = "%nbdev_export mod.file"
test_eq(is_export(cell, 'export'), (f'mod{os.path.sep}file', True))
cell['source'] = "# exporti mod.file"
test_eq(is_export(cell, 'export'), (f'mod{os.path.sep}file', False))
cell['source'] = "%nbdev_export_internal mod.file"
test_eq(is_export(cell, 'export'), (f'mod{os.path.sep}file', False))
cell['source'] = "# expt mod.file"
assert is_export(cell, 'export') is None
cell['source'] = "# exportmod.file"
assert is_export(cell, 'export') is None
cell['source'] = "# exportsmod.file"
assert is_export(cell, 'export') is None
cell['source'] = "%nbdev_export_and_showmod.file"
assert is_export(cell, 'export') is None
cell['source'] = "%nbdev_export_internalmod.file"
assert is_export(cell, 'export') is None
cell['source'] = "# exporti mod file"
assert is_export(cell, 'export') is None
cell['source'] = "%nbdev_export_internal mod file"
assert is_export(cell, 'export') is None



Find in cells the default export module.

Stops at the first cell containing a # default_exp or %nbdev_default_export flag (if there are several) and returns the value behind. Returns None if there are no cell with that code.

test_eq(find_default_export(test_nb['cells']), 'export')
assert find_default_export(test_nb['cells'][2:]) is None

Note: ReTstFlags isn't used during export but is needed by export2html and test.

class ReTstFlags[source]


Provides test flag matching regular expressions



all_flag tells us to match flags applied to all cells (True) or individual cells (False)



Return all test flags found in source[source]

Return a match object for the first test flag found in source

Listing all exported objects

The following functions make a list of everything that is exported to prepare a proper __all__ for our exported module.

tst ="""
def func(obj:Class):""")
tst, tst.groups()
(<re.Match object; span=(1, 42), match='@patch\n@log_args(a=1)\ndef func(obj:Class)'>,
 ('func', 'Class', None))


export_names(code, func_only=False)

Find the names of the objects, functions or classes defined in code that are exported.

This function only picks the zero-indented objects on the left side of an =, functions or classes (we don't want the class methods for instance) and excludes private names (that begin with _) but no dunder names. It only returns func and class names (not the objects) when func_only=True.

To work properly with fastai added python functionality, this function ignores function decorated with @typedispatch (since they are defined multiple times) and unwraps properly functions decorated with @patch.

test_eq(export_names("def my_func(x):\n  pass\nclass MyClass():"), ["my_func", "MyClass"])

#Indented funcs are ignored (funcs inside a class)
test_eq(export_names("  def my_func(x):\n  pass\nclass MyClass():"), ["MyClass"])

#Private funcs are ignored, dunder are not
test_eq(export_names("def _my_func():\n  pass\nclass MyClass():"), ["MyClass"])
test_eq(export_names("__version__ = 1:\n  pass\nclass MyClass():"), ["MyClass", "__version__"])

#trailing spaces
test_eq(export_names("def my_func ():\n  pass\nclass MyClass():"), ["my_func", "MyClass"])

#class without parenthesis
test_eq(export_names("def my_func ():\n  pass\nclass MyClass:"), ["my_func", "MyClass"])

#object and funcs
test_eq(export_names("def my_func ():\n  pass\ndefault_bla=[]:"), ["my_func", "default_bla"])
test_eq(export_names("def my_func ():\n  pass\ndefault_bla=[]:", func_only=True), ["my_func"])

#Private objects are ignored
test_eq(export_names("def my_func ():\n  pass\n_default_bla = []:"), ["my_func"])

#Objects with dots are privates if one part is private
test_eq(export_names("def my_func ():\n  pass\ndefault.bla = []:"), ["my_func", "default.bla"])
test_eq(export_names("def my_func ():\n  pass\ndefault._bla = []:"), ["my_func"])

#Monkey-path with @patch are properly renamed
test_eq(export_names("@patch\ndef my_func(x:Class):\n  pass"), ["Class.my_func"])
test_eq(export_names("@patch\ndef my_func(x:Class):\n  pass", func_only=True), ["Class.my_func"])
test_eq(export_names("some code\n@patch\ndef my_func(x:Class, y):\n  pass"), ["Class.my_func"])
test_eq(export_names("some code\n@patch\ndef my_func(x:(Class1,Class2), y):\n  pass"), ["Class1.my_func", "Class2.my_func"])

#Check delegates
test_eq(export_names("@delegates(keep=True)\nclass someClass:\n  pass"), ["someClass"])

#Typedispatch decorated functions shouldn't be added
test_eq(export_names("@patch\ndef my_func(x:Class):\n  pass\n@typedispatch\ndef func(x: TensorImage): pass"), ["Class.my_func"])

#try, except and other keywords should not be picked up (these can look like object def with type annotation)
test_eq(export_names("try:\n    a=1\nexcept:\n    b=2"), [])
test_eq(export_names("try:\n    this_might_work\nexcept:\n    b=2"), [])


extra_add(flags, code)

Catch adds to __all__ required by a cell with _all_= or %nbdev_add2all

Sometimes objects are not picked to be automatically added to the __all__ of the module so you will need to add them manually. To do so, create an exported cell with the following code %nbdev_add2all "name", "name2"

Please note:

  • elements in %nbdev_add2all can be space and/or comma separated and don't have to be quoted
    # 'func', 'func2' and 'func3' will be added to `__all__`
    %nbdev_add2all func, func2 func3
    # you'll see warnings if any unquoted function names can't be found
  • elements on a new line will not be added to __all__
    # 'func2' won't get added to `__all__`
    %nbdev_add2all 'func',
  • only the first %nbdev_add2all in a cell will get picked up
    %nbdev_add2all ['func']
    # `func2` won't get added to `__all__`
    %nbdev_add2all ['func2']
  • but you can have any number of %nbdev_add2alls in a notebook by putting them in different cells.

If you need a from __future__ import in your library, you can export your cell with special comments:

from __future__ import annotations
class ...

or magic flags:

from __future__ import annotations
class ...

Notice that %nbdev_export is after the __future__ import. Because __future__ imports must occur at the beginning of the file, nbdev allows __future__ imports in the flags section of a cell.


relative_import(name, fname)

Convert a module name to a name relative to fname

When we say from

from lib_name.module.submodule import bla

in a notebook, it needs to be converted to something like

from .module.submodule import bla

or from .submodule import bla depending on where we are. This function deals with those imports renaming.

Note that import of the form

import lib_name.module

are left as is as the syntax import module does not work for relative imports.

test_eq(relative_import('nbdev.core', Path.cwd()/'nbdev'/''), '.core')
test_eq(relative_import('nbdev.core', Path('nbdev')/'vision'/''), '..core')
test_eq(relative_import('', Path('nbdev')/'vision'/''), '.transform')
test_eq(relative_import('nbdev.notebook.core', Path('nbdev')/'data'/''), '..notebook.core')
test_eq(relative_import('', Path('nbdev')/'vision'/''), '.')

Create the library

Saving an index

To be able to build back a correspondence between functions and the notebooks they are defined in, we need to store an index. It's done in the private module _nbdev inside your library, and the following function are used to define it.



Create a skeletton for _nbdev



Reads _nbdev



Save mod inside _nbdev

Create the modules


split_flags_and_code(cell, return_type='list')

Splits the source of a cell into 2 parts and returns (flags, code)

return_type tells us if the tuple returned will contain lists of lines or strings with line breaks.

If no magic flags are found, treat the first comment line as a flag

split_flags_and_code example

If magic flags are found, the flags part can contain multiple lines

split_flags_and_code example

def _test_split_flags_and_code(expected_flags, expected_code):
    cell = nbformat.v4.new_code_cell('\n'.join(expected_flags + expected_code))
    test_eq((expected_flags, expected_code), split_flags_and_code(cell))
    expected=('\n'.join(expected_flags), '\n'.join(expected_code))
    test_eq(expected, split_flags_and_code(cell, str))
    ['# TODO: write this function',
    'def func(x): pass'])


create_mod_file(fname, nb_path)

Create a module file for fname.

A new module filename is created each time a notebook has a cell marked with %nbdev_default_export. In your collection of notebooks, you should only have one notebook that creates a given module since they are re-created each time you do a library build (to ensure the library is clean). Note that any file you create manually will never be overwritten (unless it has the same name as one of the modules defined in a %nbdev_default_export cell) so you are responsible to clean up those yourself.

fname is the notebook that contained the %nbdev_default_export cell.


create_mod_files(files, to_dict=False)

Create mod files for default exports found in files

Create module files for all %nbdev_default_export flags found in files and return a list containing the names of modules created.

Note: The number if modules returned will be less that the number of files passed in if files do not %nbdev_default_export.

By creating all module files before calling _notebook2script, the order of execution no longer matters - so you can now export to a notebook that is run "later".

You might still have problems when

  • converting a subset of notebooks or
  • exporting to a module that does not have a %nbdev_default_export yet

in which case _notebook2script will print warnings like;

Warning: Exporting to "" but this module is not part of this build

If you see a warning like this

  • and the module file (e.g. "") does not exist, you'll see a FileNotFoundError
  • if the module file exists, the exported cell will be written - even if the exported cell is already in the module file



Add in all subdirs of path containing python files if it's not there already

with tempfile.TemporaryDirectory() as d:
    os.makedirs(Path(d)/'a', exist_ok=True)
    os.makedirs(Path(d)/'a/b', exist_ok=True)
    assert not (Path(d)/'').exists()
    for e in [Path(d)/'a', Path(d)/'a/b']:
        assert (e/'').exists()



Add or update __version__ in the main of the library



Add or update baseurl in _config.yml for the docs


notebook2script(fname=None, silent=False, to_dict=False)

Convert notebooks matching fname to modules

Finds cells starting with %nbdev_export and puts them into the appropriate module. If fname is not specified, this will convert all notebook not beginning with an underscore in the nb_folder defined in setting.ini. Otherwise fname can be a single filename or a glob expression.

silent makes the command not print any statement and to_dict is used internally to convert the library to a dictionary.