merge

Fix merge conflicts in jupyter notebooks

Introduction

Git conflict markers make notebook JSON invalid. nbdev-fix creates a notebook you can open in Jupyter. It resolves metadata and output conflicts automatically. For conflicting cell contents, it keeps both versions between Markdown markers:

<<<<<< HEAD

# local code here

======

# remote code here

>>>>>> a7ec1b0bfb8e23b05fd0a2e6cafcb41cd0fb1c35

Below is an example of broken notebook. The json format is broken by the lines automatically added by git. Such a file can’t be opened in jupyter notebook.

broken = Path('../../tests/example.ipynb.broken')
tst_nb = broken.read_text(encoding='utf-8')
print(tst_nb)

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
<<<<<<< HEAD
    "z=3\n",
=======
    "z=2\n",
>>>>>>> a7ec1b0bfb8e23b05fd0a2e6cafcb41cd0fb1c35
    "z"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "6"
      ]
     },
<<<<<<< HEAD
     "execution_count": 7,
=======
     "execution_count": 5,
>>>>>>> a7ec1b0bfb8e23b05fd0a2e6cafcb41cd0fb1c35
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x=3\n",
    "y=3\n",
    "x+y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}

The second conflict affects an execution count. Either value works, so nbdev-fix can resolve it automatically. The first conflict spans two cells, including one missing from the other version. It needs a manual decision. nbdev-fix preserves both versions in a valid notebook.

Creating a merged notebook

The approach we use is to first “unpatch” the conflicted file, regenerating the two files it was originally created from. Then we redo the diff process, but using cells instead of text lines.

---

source

unpatch

def unpatch(
    s:str
):

Takes a string with conflict markers and returns the two original files, and their branch names

The result of “unpatching” our conflicted test notebook is the two original notebooks it would have been created from. Each of these original notebooks will contain valid JSON:

a,b,branch1,branch2 = unpatch(tst_nb)
dict2nb(loads(a))

{ 'cells': [ { 'cell_type': 'code',
               'execution_count': 6,
               'id': '54333cb4',
               'idx_': 0,
               'lang_': 'python',
               'metadata': {},
               'outputs': [ { 'data': {'text/plain': '3'},
                              'execution_count': 6,
                              'metadata': {},
                              'output_type': 'execute_result'}],
               'source': 'z=3\nz'},
             { 'cell_type': 'code',
               'execution_count': 5,
               'id': 'e275789e',
               'idx_': 1,
               'lang_': 'python',
               'metadata': {},
               'outputs': [ { 'data': {'text/plain': '6'},
                              'execution_count': 7,
                              'metadata': {},
                              'output_type': 'execute_result'}],
               'source': 'x=3\ny=3\nx+y'},
             { 'cell_type': 'code',
               'execution_count': None,
               'id': '15113038',
               'idx_': 2,
               'lang_': 'python',
               'metadata': {},
               'outputs': [],
               'source': ''}],
  'metadata': { 'kernelspec': { 'display_name': 'Python 3',
                                'language': 'python',
                                'name': 'python3'}},
  'nbformat': 4,
  'nbformat_minor': 2}

dict2nb(loads(b))

{ 'cells': [ { 'cell_type': 'code',
               'execution_count': 6,
               'id': 'd6c69e35',
               'idx_': 0,
               'lang_': 'python',
               'metadata': {},
               'outputs': [ { 'data': {'text/plain': '3'},
                              'execution_count': 6,
                              'metadata': {},
                              'output_type': 'execute_result'}],
               'source': 'z=2\nz'},
             { 'cell_type': 'code',
               'execution_count': 5,
               'id': '9f6fc608',
               'idx_': 1,
               'lang_': 'python',
               'metadata': {},
               'outputs': [ { 'data': {'text/plain': '6'},
                              'execution_count': 5,
                              'metadata': {},
                              'output_type': 'execute_result'}],
               'source': 'x=3\ny=3\nx+y'},
             { 'cell_type': 'code',
               'execution_count': None,
               'id': 'c8dcd223',
               'idx_': 2,
               'lang_': 'python',
               'metadata': {},
               'outputs': [],
               'source': ''}],
  'metadata': { 'kernelspec': { 'display_name': 'Python 3',
                                'language': 'python',
                                'name': 'python3'}},
  'nbformat': 4,
  'nbformat_minor': 2}

branch1,branch2

('HEAD', 'a7ec1b0bfb8e23b05fd0a2e6cafcb41cd0fb1c35')

---

source

nbdev_fix

def nbdev_fix(
    nbname:str, # Notebook filename to fix
    outname:str=None, # Filename of output notebook (defaults to `nbname`)
    nobackup:<function bool_arg at 0x7fa89fbd5120>=True, # Do not backup `nbname` to `nbname`.bak if `outname` not provided
    theirs:bool=False, # Use their outputs and metadata instead of ours
    noprint:bool=False, # Do not print info about whether conflicts are found
):

Create working notebook from conflicted notebook nbname

By default, nbdev-fix keeps local outputs and metadata. Pass theirs=True to use the other branch’s values. Conflicting source remains in separate cells. Open the repaired notebook and search for <<<<<<< to resolve these conflicts.

Without outname, the command overwrites the input notebook. Set nobackup=False to keep a .ipynb.bak copy. The completion message says whether any conflicts remain.

nbdev_fix(broken, outname='tmp.ipynb')
chk = read_nb('tmp.ipynb')
test_eq(len(chk.cells), 7)
os.unlink('tmp.ipynb')

One or more conflict remains in the notebook, please inspect manually.

Git merge driver

---

source

nbdev_merge

def nbdev_merge(
    base:str, ours:str, theirs:str, path:str
):

Git merge driver for notebooks

Run nbdev-install-hooks to install the notebook Git merge driver. It first runs Git’s standard merge, then calls nbdev-fix if conflicts remain.

Set the THEIRS environment variable to choose the other branch’s outputs and metadata:

THEIRS=True git merge branch