migrate

Utilities for migrating to nbdev

nbdev 1 –> 2

Add Alias Path To Avoid Broken Links -

When migrating from fastpages which was Jekyll based to Quarto, we want to construct aliases to mitigate broken links.

Migrate notebooks

---

source

MigrateProc

def MigrateProc(
    nb
):

Migrate fastpages front matter in notebooks to a raw cell.

Before you migrate the fastpages notebook, the front matter is specified in Markdown like this:

_tst_nb = '../../tests/2020-09-01-fastcore.ipynb'
print(read_nb(_tst_nb).cells[0].source)

# "fastcore: An Underrated Python Library"

> A unique python library that extends the python programming language and provides utilities that enhance productivity.
- author: "<a href='https://twitter.com/HamelHusain'>Hamel Husain</a>"
- toc: false
- image: images/copied_from_nb/fastcore_imgs/td.png
- comments: true
- search_exclude: true
- hide: true
- categories: [fastcore, fastai]
- permalink: /fastcore/
- badges: true

After migrating the notebook, the front matter is moved to a raw cell, and some of the fields are converted to be compliant with Quarto. Furthermore, aliases may be added in order to prevent broken links:

nbp = NBProcessor('../../tests/2020-09-01-fastcore.ipynb', procs=[FrontmatterProc, MigrateProc])
nbp.process()
_fm1 = _get_raw_fm(nbp.nb)
print(_fm1)

---
aliases:
- /fastcore/
author: <a href='https://twitter.com/HamelHusain'>Hamel Husain</a>
badges: true
categories:
- fastcore
- fastai
date: '2020-09-01'
description: A unique python library that extends the python programming language
  and provides utilities that enhance productivity.
draft: 'true'
image: fastcore_imgs/td.png
output-file: 2020-09-01-fastcore.html
permalink: /fastcore/
search: 'false'
title: 'fastcore: An Underrated Python Library'
toc: false

---

Migrate Fastpages Markdown Front Matter

---

source

fp_md_fm

def fp_md_fm(
    path
):

Make fastpages front matter in markdown files quarto compliant.

Here is what the front matter of a fastpages markdown post looks like before migration:

print(run('head -n13 ../../tests/2020-01-14-test-markdown-post.md'))

---

toc: true
layout: post
description: A minimal example of using markdown with fastpages.
categories: [markdown]
title: An Example Markdown Post


---

# Example Markdown Post

And this is what it looks like after migration:

_res = fp_md_fm('../../tests/2020-01-14-test-markdown-post.md')
print(_res[:300])

---
aliases:
- /markdown/2020/01/14/test-markdown-post
categories:
- markdown
date: '2020-01-14'
description: A minimal example of using markdown with fastpages.
layout: post
title: An Example Markdown Post
toc: true

---

# Example Markdown Post

## Basic setup

Jekyll requires blog post files to b

#hide
_res = fp_md_fm('../../tests/2022-09-06-homeschooling.md')
test_eq(_res,
"""---
aliases:
- /2022/09/06/homeschooling
author: Rachel Thomas
categories:
- advice
- health
date: '2022-09-06'
description: You can permanently damage your back, neck, and wrists from working without
  an ergonomic setup.  Learn how to create one for less at home.
image: /images/ergonomic1-short.jpg
summary: You can permanently damage your back, neck, and wrists from working without
  an ergonomic setup.  Learn how to create one for less at home.
tags: advice health
title: 'Essential Work-From-Home Advice: Cheap and Easy Ergonomic Setups'

---

Lorem ipsum
""")

Directives

nbdev v2 directives start with a #| whereas v1 directives were comments without a pipe |.

_test_dir = """
#default_exp
 #export
# collapse-show
#collapse-hide
#collapse
# collapse_output
not_dir='#export'
# hide_input
foo
# hide
"""
test_eq(_repl_directives(_test_dir),
"""
#| default_exp
#| export
#| code-fold: show
#| code-fold: true
#| code-fold: true
# collapse_output
not_dir='#export'
#| echo: false
foo
#| include: false
""")

---

source

_repl_v1dir

def _repl_v1dir(
    cell
):

Replace nbdev v1 with v2 directives.

for example, if any of the lines below are valid nbdev v1 directives, they replaced with a #|, but only before the first line of code:

Callouts

In fastpages, there was a markdown shortuct for callouts for Note, Tip, Important and Warning with block quotes (these only worked in notebooks). Since Quarto has its own callout blocks with markdown syntax, we do not implement these shortcuts in nbdev. Instead, we offer a manual conversion utility for these callouts so that you can migrate from fastpages to Quarto.

---

source

_convert_callout

def _convert_callout(
    s
):

Convert nbdev v1 to v2 callouts.

For example, the below markdown:

_callouts="""
## Boxes / Callouts

> Warning: There will be no second warning!

Other text

> Important: Pay attention! It's important.

> Tip: This is my tip.

> Note: Take note of `this.`
"""

Gets converted to:

## Boxes / Callouts

:::{.callout-warning}

There will be no second warning!

:::

Other text

:::{.callout-important}

Pay attention! It's important.

:::

Video Embeds

In fastpages, you could embed videos with a simple markdown shortcut involving a block quote with the prefix youtube:, that looked like this

> youtube: https://youtu.be/XfoYk_Z5AkI

However, in Quarto you can use the video extension to embed videos.

---

source

_convert_video

def _convert_video(
    s
):

Replace nbdev v1 with v2 video embeds.

_videos="""
## Videos

> youtube: https://youtu.be/XfoYk_Z5AkI
"""

print(_convert_video(_videos))

## Videos

---

source

migrate_nb

def migrate_nb(
    path, overwrite:bool=True
):

Migrate Notebooks from nbdev v1 and fastpages.

---

source

migrate_md

def migrate_md(
    path, overwrite:bool=True
):

Migrate Markdown Files from fastpages.

---

source

nbdev_migrate

def nbdev_migrate(
    path:str=None, # A path or glob containing notebooks and markdown files to migrate
    no_skip:bool=False, # Do not skip directories beginning with an underscore
):

Convert all markdown and notebook files in path from v1 to v2

nbdev 2 –> 3

Migrate settings.ini to pyproject.toml

---

source

nbdev_migrate_config

def nbdev_migrate_config(
    path:str='.', # Project root containing settings.ini
):

Migrate settings.ini to pyproject.toml

import tempfile

with tempfile.TemporaryDirectory() as d:
    # Create a minimal settings.ini
    (Path(d)/'settings.ini').write_text('''[DEFAULT]
repo = test-proj
user = testuser
author = Test Author
author_email = [email protected]
description = A test project
version = 1.0.0
''')
    wfd = Path(d)/'.github/workflows'
    wfd.mkdir(parents=True)
    (wfd/'deploy.yaml').write_text('      - uses: fastai/workflows/quarto-ghp@master\n')
    nbdev_migrate_config(d)
    assert (Path(d)/'pyproject.toml').exists()
    txt = (Path(d)/'pyproject.toml').read_text()
    assert 'name = "test-proj"' in txt
    assert '[tool.nbdev]' in txt
    assert (Path(d)/'test_proj/__init__.py').exists()
    wf = (Path(d)/'.github/workflows/deploy.yaml').read_text()
    assert 'uses: AnswerDotAI/workflows/quarto-ghp3@master' in wf

Created /var/folders/51/b2_szf2945n072c0vj2cyty40000gn/T/tmp64olumcw/pyproject.toml. You can now delete /var/folders/51/b2_szf2945n072c0vj2cyty40000gn/T/tmp64olumcw/settings.ini and setup.py (if present)

_test_settings_ini = '''[DEFAULT]
repo = fasthtml
lib_name = python-fasthtml
version = 0.12.40
min_python = 3.10
license = apache2
requirements = fastcore>=1.10.0 python-dateutil starlette>0.33 oauthlib itsdangerous uvicorn[standard]>=0.30
dev_requirements = ipython lxml pysym2md monsterui PyJWT
black_formatting = False
conda_user = fastai
doc_path = _docs
lib_path = fasthtml
nbs_path = nbs
recursive = True
tst_flags = notest
put_version_in_init = True
branch = main
custom_sidebar = False
doc_host = https://www.fastht.ml
doc_baseurl = /docs/
git_url = https://github.com/AnswerDotAI/fasthtml
title = fasthtml
audience = Developers
author = Jeremy Howard and contributors
author_email = [email protected]
copyright = 2024 onwards, Jeremy Howard
description = The fastest way to create an HTML app
keywords = nbdev jupyter notebook python
language = English
status = 3
console_scripts = fh_railway_link=fasthtml.cli:railway_link
    fh_railway_deploy=fasthtml.cli:railway_deploy
user = AnswerDotAI
readme_nb = index.ipynb
allowed_metadata_keys = 
allowed_cell_metadata_keys = 
jupyter_hooks = True
clean_ids = True
clear_all = False
cell_number = False
skip_procs = 
update_pyproject = True
'''

from configparser import ConfigParser
try: import tomllib
except ImportError: import tomli as tomllib

cfg = ConfigParser()
cfg.read_string(_test_settings_ini)
ini = dict2obj(dict(cfg['DEFAULT']))
first(ini.items())

('repo', 'fasthtml')

with tempfile.TemporaryDirectory() as d:
    result = _nbdev_migrate_config(dict(cfg['DEFAULT']), Path(d))
    toml = tomllib.loads(result)
proj, nbdev = toml['project'], nested_idx(toml, 'tool', 'nbdev')
proj['name']

'python-fasthtml'

proj = toml['project']
test_eq(first(nested_idx(proj, 'entry-points', 'nbdev')), ini.repo)
assert ini.repo in nested_idx(proj, 'urls', 'Repository')

test_eq(proj['name'], ini.lib_name)
test_eq(nested_idx(toml, 'tool', 'setuptools', 'dynamic', 'version', 'attr'), f"{ini.lib_path}.__version__")
test_eq(proj['requires-python'], f">={ini.min_python}")
test_eq(nested_idx(proj, 'license', 'text'), 'Apache-2.0')  # 'apache2' maps to 'Apache-2.0'
test_eq(proj['dependencies'], ini.requirements.split())
test_eq(nested_idx(proj, 'optional-dependencies', 'dev'), ini.dev_requirements.split())
test_eq(nested_idx(toml, 'tool', 'setuptools', 'packages', 'find', 'include'), [ini.lib_path])
test_eq(nested_idx(proj, 'urls', 'Documentation'), f"{ini.doc_host}{ini.doc_baseurl}")
test_eq(nested_idx(proj, 'urls', 'Repository'), ini.git_url)
nbdev = nested_idx(toml, 'tool', 'nbdev')
test_eq(nbdev['jupyter_hooks'], ini.jupyter_hooks.lower() == 'true')
# test_eq(nbdev['custom_sidebar'], ini.custom_sidebar.lower() == 'true')
test_eq(nested_idx(proj, 'authors', 0, 'name'), ini.author)
test_eq(nested_idx(proj, 'authors', 0, 'email'), ini.author_email)
test_eq(proj['description'], ini.description)
test_eq(proj['keywords'], ini.keywords.split())
test_eq(nested_idx(proj, 'scripts', 'fh_railway_link'), 'fasthtml.cli:railway_link')

scripts = ini.console_scripts.strip().split('\n')
for s in scripts:
    name,val = s.split('=')
    test_eq(nested_idx(proj, 'scripts', name.strip()), val.strip())

assert f"Development Status :: {ini.status}" in str(proj['classifiers'])
assert f"Intended Audience :: {ini.audience}" in str(proj['classifiers'])
assert f"Natural Language :: {ini.language}" in str(proj['classifiers'])

assert ini.user in nested_idx(proj, 'urls', 'Repository')

from collections import Counter

def find_dupes(d, path=''):
    res = {}
    for k,v in (d.items() if isinstance(d, dict) else enumerate(d) if isinstance(d, list) else []):
        p = f"{path}.{k}" if path else str(k)
        if isinstance(v, dict): res.update(find_dupes(v, p))
        elif isinstance(v, list):
            hashable = [x for x in v if isinstance(x, (str, int, float, bool, tuple))]
            counts = Counter(hashable)
            dupes = {x:n for x,n in counts.items() if n > 1}
            if dupes: res[p] = dupes
    return res

test_eq(find_dupes(toml), {})