# migrate
## nbdev 1 –\> 2
### Add Alias Path To Avoid Broken Links -
When migrating from fastpages which was Jekyll based to Quarto, we want
to construct aliases to mitigate broken links.
### Migrate notebooks
------------------------------------------------------------------------
source
### MigrateProc
``` python
def MigrateProc(
nb
):
```
*Migrate fastpages front matter in notebooks to a raw cell.*
Before you migrate the fastpages notebook, the front matter is specified
in Markdown like this:
``` python
_tst_nb = '../../tests/2020-09-01-fastcore.ipynb'
print(read_nb(_tst_nb).cells[0].source)
```
# "fastcore: An Underrated Python Library"
> A unique python library that extends the python programming language and provides utilities that enhance productivity.
- author: "Hamel Husain"
- toc: false
- image: images/copied_from_nb/fastcore_imgs/td.png
- comments: true
- search_exclude: true
- hide: true
- categories: [fastcore, fastai]
- permalink: /fastcore/
- badges: true
After migrating the notebook, the front matter is moved to a raw cell,
and some of the fields are converted to be compliant with Quarto.
Furthermore, aliases may be added in order to prevent broken links:
``` python
nbp = NBProcessor('../../tests/2020-09-01-fastcore.ipynb', procs=[FrontmatterProc, MigrateProc])
nbp.process()
_fm1 = _get_raw_fm(nbp.nb)
print(_fm1)
```
---
aliases:
- /fastcore/
author: Hamel Husain
badges: true
categories:
- fastcore
- fastai
date: '2020-09-01'
description: A unique python library that extends the python programming language
and provides utilities that enhance productivity.
draft: 'true'
image: fastcore_imgs/td.png
output-file: 2020-09-01-fastcore.html
permalink: /fastcore/
search: 'false'
title: 'fastcore: An Underrated Python Library'
toc: false
---
### Migrate Fastpages Markdown Front Matter
------------------------------------------------------------------------
source
### fp_md_fm
``` python
def fp_md_fm(
path
):
```
*Make fastpages front matter in markdown files quarto compliant.*
Here is what the front matter of a fastpages markdown post looks like
before migration:
``` python
print(run('head -n13 ../../tests/2020-01-14-test-markdown-post.md'))
```
---
toc: true
layout: post
description: A minimal example of using markdown with fastpages.
categories: [markdown]
title: An Example Markdown Post
---
# Example Markdown Post
And this is what it looks like after migration:
``` python
_res = fp_md_fm('../../tests/2020-01-14-test-markdown-post.md')
print(_res[:300])
```
---
aliases:
- /markdown/2020/01/14/test-markdown-post
categories:
- markdown
date: '2020-01-14'
description: A minimal example of using markdown with fastpages.
layout: post
title: An Example Markdown Post
toc: true
---
# Example Markdown Post
## Basic setup
Jekyll requires blog post files to b
``` python
#hide
_res = fp_md_fm('../../tests/2022-09-06-homeschooling.md')
test_eq(_res,
"""---
aliases:
- /2022/09/06/homeschooling
author: Rachel Thomas
categories:
- advice
- health
date: '2022-09-06'
description: You can permanently damage your back, neck, and wrists from working without
an ergonomic setup. Learn how to create one for less at home.
image: /images/ergonomic1-short.jpg
summary: You can permanently damage your back, neck, and wrists from working without
an ergonomic setup. Learn how to create one for less at home.
tags: advice health
title: 'Essential Work-From-Home Advice: Cheap and Easy Ergonomic Setups'
---
Lorem ipsum
""")
```
### Directives
nbdev v2 directives start with a `#|` whereas v1 directives were
comments without a pipe `|`.
``` python
_test_dir = """
#default_exp
#export
# collapse-show
#collapse-hide
#collapse
# collapse_output
not_dir='#export'
# hide_input
foo
# hide
"""
test_eq(_repl_directives(_test_dir),
"""
#| default_exp
#| export
#| code-fold: show
#| code-fold: true
#| code-fold: true
# collapse_output
not_dir='#export'
#| echo: false
foo
#| include: false
""")
```
------------------------------------------------------------------------
source
### \_repl_v1dir
``` python
def _repl_v1dir(
cell
):
```
*Replace nbdev v1 with v2 directives.*
for example, if any of the lines below are valid nbdev v1 directives,
they replaced with a `#|`, but only before the first line of code:
### Callouts
In fastpages, there was a markdown shortuct for callouts for `Note`,
`Tip`, `Important` and `Warning` with block quotes (these only worked in
notebooks). Since Quarto has its own [callout
blocks](https://quarto.org/docs/authoring/callouts.html#callout-types)
with markdown syntax, we do not implement these shortcuts in nbdev.
Instead, we offer a manual conversion utility for these callouts so that
you can migrate from fastpages to Quarto.
------------------------------------------------------------------------
source
### \_convert_callout
``` python
def _convert_callout(
s
):
```
*Convert nbdev v1 to v2 callouts.*
For example, the below markdown:
``` python
_callouts="""
## Boxes / Callouts
> Warning: There will be no second warning!
Other text
> Important: Pay attention! It's important.
> Tip: This is my tip.
> Note: Take note of `this.`
"""
```
Gets converted to:
## Boxes / Callouts
:::{.callout-warning}
There will be no second warning!
:::
Other text
:::{.callout-important}
Pay attention! It's important.
:::
### Video Embeds
In fastpages, you could embed videos with a simple markdown shortcut
involving a block quote with the prefix `youtube:`, that looked like
this
`> youtube: https://youtu.be/XfoYk_Z5AkI`
However, in Quarto you can use the [video
extension](https://github.com/quarto-ext/video) to embed videos.
------------------------------------------------------------------------
source
### \_convert_video
``` python
def _convert_video(
s
):
```
*Replace nbdev v1 with v2 video embeds.*
``` python
_videos="""
## Videos
> youtube: https://youtu.be/XfoYk_Z5AkI
"""
```
``` python
print(_convert_video(_videos))
```
## Videos
https://youtu.be/XfoYk_Z5AkI
------------------------------------------------------------------------
source
### migrate_nb
``` python
def migrate_nb(
path, overwrite:bool=True
):
```
*Migrate Notebooks from nbdev v1 and fastpages.*
------------------------------------------------------------------------
source
### migrate_md
``` python
def migrate_md(
path, overwrite:bool=True
):
```
*Migrate Markdown Files from fastpages.*
------------------------------------------------------------------------
source
### nbdev_migrate
``` python
def nbdev_migrate(
path:str=None, # A path or glob containing notebooks and markdown files to migrate
no_skip:bool=False, # Do not skip directories beginning with an underscore
):
```
*Convert all markdown and notebook files in `path` from v1 to v2*
## nbdev 2 –\> 3
### Migrate settings.ini to pyproject.toml
------------------------------------------------------------------------
source
### nbdev_migrate_config
``` python
def nbdev_migrate_config(
path:str='.', # Project root containing settings.ini
):
```
*Migrate settings.ini to pyproject.toml*
``` python
import tempfile
```
``` python
with tempfile.TemporaryDirectory() as d:
# Create a minimal settings.ini
(Path(d)/'settings.ini').write_text('''[DEFAULT]
repo = test-proj
user = testuser
author = Test Author
author_email = test@test.com
description = A test project
version = 1.0.0
''')
nbdev_migrate_config(d)
assert (Path(d)/'pyproject.toml').exists()
txt = (Path(d)/'pyproject.toml').read_text()
assert 'name = "test-proj"' in txt
assert '[tool.nbdev]' in txt
assert (Path(d)/'test_proj/__init__.py').exists()
```
Created /var/folders/51/b2_szf2945n072c0vj2cyty40000gn/T/tmpv1z3p_34/pyproject.toml. You can now delete /var/folders/51/b2_szf2945n072c0vj2cyty40000gn/T/tmpv1z3p_34/settings.ini and setup.py (if present)
``` python
_test_settings_ini = '''[DEFAULT]
repo = fasthtml
lib_name = python-fasthtml
version = 0.12.40
min_python = 3.10
license = apache2
requirements = fastcore>=1.10.0 python-dateutil starlette>0.33 oauthlib itsdangerous uvicorn[standard]>=0.30
dev_requirements = ipython lxml pysymbol_llm monsterui PyJWT
black_formatting = False
conda_user = fastai
doc_path = _docs
lib_path = fasthtml
nbs_path = nbs
recursive = True
tst_flags = notest
put_version_in_init = True
branch = main
custom_sidebar = False
doc_host = https://www.fastht.ml
doc_baseurl = /docs/
git_url = https://github.com/AnswerDotAI/fasthtml
title = fasthtml
audience = Developers
author = Jeremy Howard and contributors
author_email = github@jhoward.fastmail.fm
copyright = 2024 onwards, Jeremy Howard
description = The fastest way to create an HTML app
keywords = nbdev jupyter notebook python
language = English
status = 3
console_scripts = fh_railway_link=fasthtml.cli:railway_link
fh_railway_deploy=fasthtml.cli:railway_deploy
user = AnswerDotAI
readme_nb = index.ipynb
allowed_metadata_keys =
allowed_cell_metadata_keys =
jupyter_hooks = True
clean_ids = True
clear_all = False
cell_number = False
skip_procs =
update_pyproject = True
'''
```
``` python
from configparser import ConfigParser
try: import tomllib
except ImportError: import tomli as tomllib
```
``` python
cfg = ConfigParser()
cfg.read_string(_test_settings_ini)
ini = dict2obj(dict(cfg['DEFAULT']))
first(ini.items())
```
('repo', 'fasthtml')
``` python
with tempfile.TemporaryDirectory() as d:
result = _nbdev_migrate_config(dict(cfg['DEFAULT']), Path(d))
toml = tomllib.loads(result)
proj, nbdev = toml['project'], nested_idx(toml, 'tool', 'nbdev')
proj['name']
```
'python-fasthtml'
``` python
proj = toml['project']
test_eq(first(nested_idx(proj, 'entry-points', 'nbdev')), ini.repo)
assert ini.repo in nested_idx(proj, 'urls', 'Repository')
```
``` python
test_eq(proj['name'], ini.lib_name)
test_eq(nested_idx(toml, 'tool', 'setuptools', 'dynamic', 'version', 'attr'), f"{ini.lib_path}.__version__")
test_eq(proj['requires-python'], f">={ini.min_python}")
test_eq(nested_idx(proj, 'license', 'text'), 'Apache-2.0') # 'apache2' maps to 'Apache-2.0'
test_eq(proj['dependencies'], ini.requirements.split())
test_eq(nested_idx(proj, 'optional-dependencies', 'dev'), ini.dev_requirements.split())
test_eq(nested_idx(toml, 'tool', 'setuptools', 'packages', 'find', 'include'), [ini.lib_path])
test_eq(nested_idx(proj, 'urls', 'Documentation'), f"{ini.doc_host}{ini.doc_baseurl}")
test_eq(nested_idx(proj, 'urls', 'Repository'), ini.git_url)
nbdev = nested_idx(toml, 'tool', 'nbdev')
test_eq(nbdev['jupyter_hooks'], ini.jupyter_hooks.lower() == 'true')
# test_eq(nbdev['custom_sidebar'], ini.custom_sidebar.lower() == 'true')
test_eq(nested_idx(proj, 'authors', 0, 'name'), ini.author)
test_eq(nested_idx(proj, 'authors', 0, 'email'), ini.author_email)
test_eq(proj['description'], ini.description)
test_eq(proj['keywords'], ini.keywords.split())
test_eq(nested_idx(proj, 'scripts', 'fh_railway_link'), 'fasthtml.cli:railway_link')
```
``` python
scripts = ini.console_scripts.strip().split('\n')
for s in scripts:
name,val = s.split('=')
test_eq(nested_idx(proj, 'scripts', name.strip()), val.strip())
assert f"Development Status :: {ini.status}" in str(proj['classifiers'])
assert f"Intended Audience :: {ini.audience}" in str(proj['classifiers'])
assert f"Natural Language :: {ini.language}" in str(proj['classifiers'])
assert ini.user in nested_idx(proj, 'urls', 'Repository')
```
``` python
from collections import Counter
```
``` python
def find_dupes(d, path=''):
res = {}
for k,v in (d.items() if isinstance(d, dict) else enumerate(d) if isinstance(d, list) else []):
p = f"{path}.{k}" if path else str(k)
if isinstance(v, dict): res.update(find_dupes(v, p))
elif isinstance(v, list):
hashable = [x for x in v if isinstance(x, (str, int, float, bool, tuple))]
counts = Counter(hashable)
dupes = {x:n for x,n in counts.items() if n > 1}
if dupes: res[p] = dupes
return res
test_eq(find_dupes(toml), {})
```