The functions that transform the dev notebooks in the documentation of the library

The most important function defined in this module is notebook2html, so you may want to jump to it before scrolling though the rest, which explain the details behind the scenes of the conversion from notebooks to the html documentation. The main things to remember are:

  • put a #hide or %nbdev_hide flag at the top of any cell you want to completely hide in the docs
  • use the hide input jupyter extension to hide the input of some cells (by default all show_doc cells have that marker added)
  • you can define some jekyll metadata in the markdown cell with the title, see get_metadata
  • use backsticks for terms you want automatic links to be found, but use <code> and </code> when you have homonyms and don't want those links
  • you can define the default toc level of classes with %nbdev_default_class_level flag followed by a number (default is 2)
  • you can add jekyll warnings, important or note banners with appropriate block quotes (see add_jekyll_notes)
  • put any images you want to use in the images folder of your notebook folder, they will be automatically copied over to the docs folder
  • put a #hide_input or %nbdev_hide_input flag at the top of a cell if you don't want code to be shown in the docs
    • cells containing %nbdev_export or show_doc have their code hidden automatically
  • put a #hide_output or %nbdev_hide_output flag at the top of a cell if you don't want output to be shown in the docs
  • use %nbdev_collapse_input or %nbdev_collapse_output to include code or output in the docs under a collapsable element

Preprocessing notebook

Cell processors

class HTMLParseAttrs[source]

HTMLParseAttrs(convert_charrefs=True) :: HTMLParser

Simple HTML parser which stores any attributes in attrs dict

h = HTMLParseAttrs()
t = h('<img src="src" alt="alt" width="700" caption="cap" />')
test_eq(t['width'], '700')
test_eq(t['src'  ], 'src')
t['width'] = '600'
test_eq(h.show(), '<img src="src" alt="alt" width="600" caption="cap" />')
t['max-width'] = t.pop('width')
test_eq(h.show(), '<img src="src" alt="alt" caption="cap" max-width="600" />')

The following functions are applied on individual cells as a preprocessing step before the conversion to html.

remove_widget_state[source]

remove_widget_state(cell)

Remove widgets in the output of cells

Those outputs usually can't be rendered properly in html.

upd_metadata[source]

upd_metadata(cell, key, value=True)

Sets key to value on the metadata of cell without replacing metadata

hide_cells[source]

hide_cells(cell)

Hide inputs of cell that need to be hidden

This concerns all the cells with %nbdev_export or %nbdev_hide_input flags and all the cell containing a show_doc for a function or class.

for source in ['show_doc(read_nb)', '# export\nfrom local.core import *', '# hide_input\n2+2',
              'line1\n show_doc (read_nb) \nline3', '# export with.mod\nfrom local.core import *',
              '%nbdev_export with.mod\nfrom local.core import *', 'line1\n%nbdev_export\nline3',
              '%nbdev_show_doc read_nb', '%nbdev_hide_input\n2+2']:
    cell = {'cell_type': 'code', 'source': source}
    cell1 = hide_cells(cell.copy())
    assert 'metadata' in cell1
    assert 'hide_input' in cell1['metadata']
    assert cell1['metadata']['hide_input']

for flag in ['# exports', '%nbdev_export_and_show']:
    cell = {'cell_type': 'code', 'source': f'{flag}\nfrom local.core2 import *'}
    test_eq(hide_cells(cell.copy()), cell)

This concerns all the cells with a %nbdev_hide_output flag.

for source in ['# hide-output\nfrom local.core import *', '# hide_output\n2+2', '%nbdev_hide_output\na=b']:
    cell = {'cell_type': 'code', 'source': source}
    cell1 = hide_cells(cell.copy())
    assert 'metadata' in cell1
    assert 'hide_output' in cell1['metadata']
    assert cell1['metadata']['hide_output']

cell = {'cell_type': 'code', 'source': '# hide-outputs\nfrom local.core import *'}
test_eq(hide_cells(cell.copy()), cell)

clean_exports[source]

clean_exports(cell)

Remove all flags from code cells

The rest of the cell is displayed without any modification.

for flag in ['# exports', '%nbdev_export_and_show']:
    cell = {'cell_type': 'code', 'source': f'{flag}\nfrom local.core import *'}
    test_eq(clean_exports(cell.copy()), {'cell_type': 'code', 'source': 'from local.core import *'})
    cell['cell_type'] = 'markdown'
    test_eq(clean_exports(cell.copy()), cell)
    cell = {'cell_type': 'code', 'source': f'{flag} core\nfrom local.core import *'}
    test_eq(clean_exports(cell.copy()), {'cell_type': 'code', 'source': 'from local.core import *'})
    
cell = {'cell_type': 'code', 'source': f'# comment \n# exports\nprint("something")'}
test_eq(clean_exports(cell.copy()), {'cell_type': 'code', 'source': '# exports\nprint("something")'})
cell = {'cell_type': 'code', 'source': f'# comment \n%nbdev_export_and_show\nprint("something")'}
test_eq(clean_exports(cell.copy()), {'cell_type': 'code', 'source': 'print("something")'})

treat_backticks[source]

treat_backticks(cell)

Add links to backticks words in cell

cell = {'cell_type': 'markdown', 'source': 'This is a `DocsTestClass`'}
test_eq(treat_backticks(cell), {'cell_type': 'markdown',
    'source': 'This is a [`DocsTestClass`](/export#DocsTestClass)'})

add_jekyll_notes[source]

add_jekyll_notes(cell)

Convert block quotes to jekyll notes in cell

Supported styles are Warning, Note Tip and Important:

Typing > Warning: There will be no second warning! will render in the docs:

Typing > Important: Pay attention! It's important. will render in the docs: Typing > Tip: This is my tip. will render in the docs: Typing > Note: Take note of this. will render in the docs: Typing > Note: A doc link to [`add_jekyll_notes`](/export2html#add_jekyll_notes) should also work fine. will render in the docs:

m=_re_image.search('![Alt](images/logo.png)')
test_eq(m.groups(), ('![Alt](', 'images/logo.png', ')', None))
# using ) or whitespace to close the group means we don't need a special case for captions
m=_re_image.search('![Alt](images/logo.png "caption (something)")')
test_eq(m.groups(), ('![Alt](', 'images/logo.png', '', None))

copy_images[source]

copy_images(cell, fname, dest, jekyll=True)

Copy images referenced in cell from fname parent folder to dest folder

This is to ensure that all images defined in nbs_folder/images and used in notebooks are copied over to doc_folder/images.

dest_img = Config().doc_path/'images'/'logo.png'
cell = {'cell_type': 'markdown', 'source':'Text\n![Alt](images/logo.png)'}
try:
    copy_images(cell, Path('01_export.ipynb'), Config().doc_path)
    test_eq(cell["source"], 'Text\n![Alt](/images/logo.png)')
    #Image has been copied
    assert dest_img.exists()
    cell = {'cell_type': 'markdown', 'source':'Text\n![Alt](images/logo.png "caption (something)")'}
    copy_images(cell, Path('01_export.ipynb'), Config().doc_path)
    test_eq(cell["source"], 'Text\n![Alt](/images/logo.png "caption (something)")')
finally: dest_img.unlink()

adapt_img_path[source]

adapt_img_path(cell, fname, dest, jekyll=True)

Adapt path of images referenced in cell from fname to work in folder dest

This function is slightly different as it ensures that a notebook convert to a file that will be placed in dest will have the images location updated. It is used for the README.md file (generated automatically from the index) since the images are copied inside the github repo, but in general, you should make sure your images are going to be accessible from the location your file ends up being.

cell = {'cell_type': 'markdown', 'source': 'Text\n![Alt](images/logo.png)'}
cell1 = adapt_img_path(cell, Path('01_export.ipynb'), Path('.').absolute().parent)
test_eq(cell1['source'], 'Text\n![Alt](nbs/images/logo.png)')

cell = {'cell_type': 'markdown', 'source': 'Text\n![Alt](http://site.logo.png)'}
cell1 = adapt_img_path(cell, Path('01_export.ipynb'), Path('.').absolute().parent)
test_eq(cell1['source'], 'Text\n![Alt](http://site.logo.png)')

cell = {'cell_type': 'markdown', 
        'source': 'Text\n<img alt="Logo" src="images/logo.png" width="600"/>'}
cell1 = adapt_img_path(cell, Path('01_export.ipynb'), Path('.').absolute().parent)
test_eq(cell1['source'], 'Text\n{% include image.html alt="Logo" max-width="600" file="nbs/images/logo.png" %}')

cell = {'cell_type': 'markdown', 
        'source': 'Text\n<img alt="Logo" src="https://site.image.png" width="600"/>'}
cell1 = adapt_img_path(cell, Path('01_export.ipynb'), Path('.').absolute().parent)
test_eq(cell1['source'], 'Text\n{% include image.html alt="Logo" max-width="600" file="https://site.image.png" %}')

Escape Latex in liquid

escape_latex[source]

escape_latex(cell)

cell = {'cell_type': 'markdown', 
        'source': 'lala\n$$equation$$\nlala'}
cell = escape_latex(cell)
test_eq(cell['source'], 'lala\n{% raw %}\n$$equation$$\n{% endraw %}\nlala')

Collapsable Code Cells

collapse_cells[source]

collapse_cells(cell)

Add a collapse button to inputs or outputs of cell in either the open or closed position

  • Placing %nbdev_collapse_input open in a code cell will inlcude your code under a collapsable element that is open by default.
print('This code cell is not collapsed by default but you can collapse it to hide it from view!')
print("Note that the output always shows with `%nbdev_collapse_input`.")
This code cell is not collapsed by default but you can collapse it to hide it from view!
Note that the output always shows with `%nbdev_collapse_input`.
  • Placing %nbdev_collapse_input in a code cell will include your code in a collapsable element that is closed by default. For example:
print('The code cell that produced this output is collapsed by default but you can expand it!')
The code cell that produced this output is collapsed by default but you can expand it!
  • Placing %nbdev_collapse_output in a code cell will hide the output under a collapsable element that is closed by default.
print('The input of this cell is visible as usual.\nHowever, the OUTPUT of this cell is collapsed by default but you can expand it!')
The input of this cell is visible as usual.
However, the OUTPUT of this cell is collapsed by default but you can expand it!

Preprocessing the list of cells

The following functions are applied to the entire list of cells of the notebook as a preprocessing step before the conversion to html.

remove_hidden[source]

remove_hidden(cells)

Remove in cells the ones with a flag #hide, #default_exp, #default_cls_lvl or #exporti

cells = [{'cell_type': 'code', 'source': source, 'hide': hide} for hide, source in [
    (False, '# export\nfrom local.core import *'), 
    (True, '# export\n%nbdev_default_class_level 4\nfrom local.core import *'), 
    (True, ' # default_cls_lvl 2 \n# export\nfrom local.core import *'), 
    (False, '%nbdev_export \nfrom local.core import *'), 
    (False, '# exporti mod file'), # Note: this used to get removed but we're more strict now
    (False, '%nbdev_export_internal mod file'),
    (True, '# hide\nfrom local.core import *'),
    (True, '%nbdev_hide\nfrom local.core import *'),
    (False, '# hide_input\nfrom local.core import *'),
    (False, '%nbdev_hide_output\nfrom local.core import *'),
    (False, '#exports\nsuper code'),
    (False, '%nbdev_export_and_show\nsuper code'),
    (True, '#default_exp notebook.export'),
    (True, '%nbdev_default_export notebook.export'),
    (False, 'show_doc(read_nb)'),
    (True, '#default_cls_lvl 3'),
    (False, '#all_slow'), # slow is not in setting.ini tst_flags
    (False, '%nbdev_slow_test all'),
    (True, '#all_fastai2'),
    (True, '%nbdev_fastai2_test all'),
    (False, '#hide (last test of to_concat)'),
    (True, '# exporti\n1 + 1'),
    (True, '%nbdev_export_internal\n1 + 1')]] + [
    {'cell_type': 'markdown', 'source': source, 'hide': hide} for hide, source in [
    (False, '#hide_input\nnice'), 
    (False, '%nbdev_hide_input\nnice'), 
    (True, '#hide\n\nto hide'), 
    (True, '#comment\n%nbdev_hide\nto hide')]]

test_eq([cell for cell in cells if not cell['hide']], remove_hidden(cells))

find_default_level[source]

find_default_level(cells)

Find in cells the default class level.

tst_nb = read_nb('00_export.ipynb')
test_eq(find_default_level(tst_nb['cells']), 3)

add_show_docs[source]

add_show_docs(cells, cls_lvl=None)

Add show_doc for each exported function or class

This only adds cells with a show_doc for non-documented functions, so if you add yourself a show_doc cell (because you want to change one of the default argument), there won't be any duplicates.

for i,cell in enumerate(tst_nb['cells']):
    if cell['source'].startswith('%nbdev_export\ndef read_nb'): break
tst_cells = [c.copy() for c in tst_nb['cells'][i-1:i+1]]
added_cells = add_show_docs(tst_cells, cls_lvl=3)
test_eq(len(added_cells), 3)
test_eq(added_cells[0], tst_nb['cells'][i-1])
test_eq(added_cells[1], tst_nb['cells'][i])
test_eq(added_cells[2], _show_doc_cell('read_nb', cls_lvl=3))
test_eq(added_cells[2]['source'], 'show_doc(read_nb, default_cls_level=3)')

for flag in ['#export', '%nbdev_export', '#exports', '%nbdev_export_and_show']:
    for show_doc_source in [
            ('show_doc(my_func)', 'show_doc(my_func, title_level=3)'),
            ('%nbdev_show_doc my_func', '%nbdev_show_doc my_func, title_level=3')]:
        #Check show_doc isn't added if it was already there.
        tst_cells1 = [{'cell_type':'code', 'source': f'{flag}\ndef my_func(x):\n    return x'},
                      {'cell_type':'code', 'source': show_doc_source[0]}]
        test_eq(add_show_docs(tst_cells1), tst_cells1)
        #Check show_doc is added
        test_eq(len(add_show_docs(tst_cells1[:-1])), len(tst_cells1))
        tst_cells1 = [{'cell_type':'code', 'source': f'{flag} with.mod\ndef my_func(x):\n    return x'},
                      {'cell_type':'markdown', 'source': 'Some text'},
                      {'cell_type':'code', 'source': show_doc_source[1]}]
        test_eq(add_show_docs(tst_cells1), tst_cells1)
        #Check show_doc is added when using mod export
        test_eq(len(add_show_docs(tst_cells1[:-1])), len(tst_cells1))

remove_fake_headers[source]

remove_fake_headers(cells)

Remove in cells the fake header

You can fake headers in your notebook to navigate them more easily with collapsible headers, just make them finish with a dash and they will be removed. One typical use case is to have a header of level 2 with the name of a class, since the show_doc cell of that class will create the same anchor, you need to have the one you created manually disappear to avoid any duplicate.

cells = [{'cell_type': 'markdown',
          'metadata': {},
          'source': '### Fake-'}] + tst_nb['cells'][:10]
cells1 = remove_fake_headers(cells)
test_eq(len(cells1), len(cells)-1)
test_eq(cells1[0], cells[1])

remove_empty[source]

remove_empty(cells)

Remove in cells the empty cells

Grabbing metada

get_metadata[source]

get_metadata(cells)

Find the cell with title and summary in cells.

In the markdown cell with the title, you can add the summary as a block quote (just put an empty block quote for an empty summary) and a list with any additional metadata you would like to add, for instance:

# Title
> Awesome summary
- toc:False

The toc: False metadata will prevent the table of contents from showing on the page.

tst_nb = read_nb('00_export.ipynb')
test_eq(get_metadata(tst_nb['cells']), {
    'keywords': 'fastai',
    'summary': 'The functions that transform notebooks in a library',
    'title': 'Export to modules'})

#The cell with the metada is popped out, so if we do it a second time we get the default.
test_eq(get_metadata(tst_nb['cells']), {'keywords': 'fastai', 'title'   : 'Title'})

Executing show_doc cells

class ExecuteShowDocPreprocessor[source]

ExecuteShowDocPreprocessor(*args, **kwargs) :: ExecutePreprocessor

An ExecutePreprocessor that only executes show_doc and import cells

Cells containing:

are not run while building docs. This avoids failures caused by importing empty or partially built modules.

Cells containing:

  • show_doc (which could be indented) or
  • a "library import" (zero indent import from current library) e.g. from LIB_NAME.core import *

are executed and must run without error. If running these cells raises an exception, the build will stop.

Cells containing zero indented imports. e.g.

  • from module import * or
  • import module

are executed but errors will not stop the build.

If you need to show_doc something, please make sure it's imported via a cell that does not depend on previous cells being run. The easiest way to do this is to use a cell that contains nothing but imports.

execute_nb[source]

execute_nb(nb, mod=None, metadata=None, show_doc_only=True)

Execute nb (or only the show_doc cells) with metadata

Converting bibtex citations

cite2link(cell)

Creates links from \cite{} to Refenrence section generated by jupyter_latex_envs

jupyter_latex_envs is a jupyter extension https://github.com/jfbercher/jupyter_latex_envs.

You can find relevant section here Note, that nbdev now only supports [<a class="latex_cit" id="call-" href="#cit-"></a>] conversion and not the rest, e.g., \figure{} and so on.

It's important to execute all show_doc cells before exporting the notebook to html because some of them have just been added automatically or others could have outdated links.

fake_nb = {k:v for k,v in tst_nb.items() if k != 'cells'}
fake_nb['cells'] = [tst_nb['cells'][0].copy()] + added_cells
fake_nb = execute_nb(fake_nb, mod='export')
assert len(fake_nb['cells'][-1]['outputs']) > 0

Filling templates

The following functions automatically adds jekyll templates if they are misssing.

write_tmpl[source]

write_tmpl(tmpl, nms, cfg, dest)

Write tmpl to dest (if missing) filling in nms in template using dict cfg

write_tmpls[source]

write_tmpls()

Write out _config.yml and _data/topnav.yml using templates

Conversion

__file__ = Config().lib_path/'export2html.py'

nbdev_exporter[source]

nbdev_exporter(template_file=None)

convert_nb[source]

convert_nb(fname, template_file=None, exporter=None, dest=None)

Convert a notebook fname to html file in dest_path.

notebook2html[source]

notebook2html(fname=None, force_all=False, n_workers=None, template_file=None, exporter=None, dest=None)

Convert all notebooks matching fname to html files

Hide cells starting with %nbdev_export and only leaves the prose and the tests. If fname is not specified, this will convert all notebooks not beginning with an underscore in the nb_folder defined in setting.ini. Otherwise fname can be a single filename or a glob expression.

By default, only the notebooks that are more recent than their html counterparts are modified, pass force_all=True to change that behavior.

convert_md[source]

convert_md(fname, dest_path, img_path='docs/images/', jekyll=True)

Convert a notebook fname to a markdown file in dest_path.

This is used to convert the index into the README.md.

t = '![screenshot](attachment:image.png)'
test_eq(_re_att_ref.match(t).groups(), ('screenshot', None))

t = '![screenshot](attachment:image.png "Deploying to Binder")'
test_eq(_re_att_ref.match(t).groups(), ('screenshot', "Deploying to Binder"))

nb_detach_cells[source]

nb_detach_cells(path_nb, dest=None, replace=True, use_img=False)

Export cell attachments to dest and update references

create_default_sidebar[source]

create_default_sidebar()

Create the default sidebar for the docs website

The default sidebar lists all html pages with their respective title, except the index that is named "Overview". To build a custom sidebar, set the flag custom_sidebar in your settings.ini to True then change the sidebar.json file in the doc_folder to your liking. Otherwise, the sidebar is updated at each doc build.

make_sidebar[source]

make_sidebar()

Making sidebar for the doc website form the content of doc_folder/sidebar.json