merge

Fix merge conflicts in jupyter notebooks

Introduction

When working with jupyter notebooks (which are json files behind the scenes) and GitHub, it is very common that a merge conflict (that will add new lines in the notebook source file) will break some notebooks you are working on. This module defines the function nbdev_fix to fix those notebooks for you, and attempt to automatically merge standard conflicts. The remaining ones will be delimited by markdown cells like this:

<<<<<< HEAD

# local code here

======

# remote code here

>>>>>> a7ec1b0bfb8e23b05fd0a2e6cafcb41cd0fb1c35

Below is an example of broken notebook. The json format is broken by the lines automatically added by git. Such a file can’t be opened in jupyter notebook.

broken = Path('../../tests/example.ipynb.broken')
tst_nb = broken.read_text(encoding='utf-8')
print(tst_nb)

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
<<<<<<< HEAD
    "z=3\n",
=======
    "z=2\n",
>>>>>>> a7ec1b0bfb8e23b05fd0a2e6cafcb41cd0fb1c35
    "z"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "6"
      ]
     },
<<<<<<< HEAD
     "execution_count": 7,
=======
     "execution_count": 5,
>>>>>>> a7ec1b0bfb8e23b05fd0a2e6cafcb41cd0fb1c35
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x=3\n",
    "y=3\n",
    "x+y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}

Note that in this example, the second conflict is easily solved: it just concerns the execution count of the second cell and can be solved by choosing either option without really impacting your notebook. This is the kind of conflict we will fix automatically. The first conflict is more complicated as it spans across two cells and there is a cell present in one version, not the other. Such a conflict (and generally the ones where the inputs of the cells change form one version to the other) aren’t automatically fixed, but we will return a proper json file where the annotations introduced by git will be placed in markdown cells.

Creating a merged notebook

The approach we use is to first “unpatch” the conflicted file, regenerating the two files it was originally created from. Then we redo the diff process, but using cells instead of text lines.

---

source

unpatch

 unpatch (s:str)

Takes a string with conflict markers and returns the two original files, and their branch names

The result of “unpatching” our conflicted test notebook is the two original notebooks it would have been created from. Each of these original notebooks will contain valid JSON:

a,b,branch1,branch2 = unpatch(tst_nb)
dict2nb(loads(a))

{ 'cells': [ { 'cell_type': 'code',
               'execution_count': 6,
               'idx_': 0,
               'metadata': {},
               'outputs': [ { 'data': {'text/plain': ['3']},
                              'execution_count': 6,
                              'metadata': {},
                              'output_type': 'execute_result'}],
               'source': 'z=3\nz'},
             { 'cell_type': 'code',
               'execution_count': 5,
               'idx_': 1,
               'metadata': {},
               'outputs': [ { 'data': {'text/plain': ['6']},
                              'execution_count': 7,
                              'metadata': {},
                              'output_type': 'execute_result'}],
               'source': 'x=3\ny=3\nx+y'},
             { 'cell_type': 'code',
               'execution_count': None,
               'idx_': 2,
               'metadata': {},
               'outputs': [],
               'source': ''}],
  'metadata': { 'kernelspec': { 'display_name': 'Python 3',
                                'language': 'python',
                                'name': 'python3'}},
  'nbformat': 4,
  'nbformat_minor': 2}

dict2nb(loads(b))

{ 'cells': [ { 'cell_type': 'code',
               'execution_count': 6,
               'idx_': 0,
               'metadata': {},
               'outputs': [ { 'data': {'text/plain': ['3']},
                              'execution_count': 6,
                              'metadata': {},
                              'output_type': 'execute_result'}],
               'source': 'z=2\nz'},
             { 'cell_type': 'code',
               'execution_count': 5,
               'idx_': 1,
               'metadata': {},
               'outputs': [ { 'data': {'text/plain': ['6']},
                              'execution_count': 5,
                              'metadata': {},
                              'output_type': 'execute_result'}],
               'source': 'x=3\ny=3\nx+y'},
             { 'cell_type': 'code',
               'execution_count': None,
               'idx_': 2,
               'metadata': {},
               'outputs': [],
               'source': ''}],
  'metadata': { 'kernelspec': { 'display_name': 'Python 3',
                                'language': 'python',
                                'name': 'python3'}},
  'nbformat': 4,
  'nbformat_minor': 2}

branch1,branch2

('HEAD', 'a7ec1b0bfb8e23b05fd0a2e6cafcb41cd0fb1c35')

---

source

nbdev_fix

 nbdev_fix (nbname:str, outname:str=None, nobackup:<function
            bool_arg>=True, theirs:bool=False, noprint:bool=False)

Create working notebook from conflicted notebook nbname

	Type	Default	Details
nbname	str		Notebook filename to fix
outname	str	None	Filename of output notebook (defaults to nbname)
nobackup	bool_arg	True	Do not backup nbname to nbname.bak if outname not provided
theirs	bool	False	Use their outputs and metadata instead of ours
noprint	bool	False	Do not print info about whether conflicts are found

This begins by optionally backing the notebook fname to fname.bak in case something goes wrong. Then it parses the broken json, solving conflicts in cells. Every conflict that only involves metadata or outputs of cells will be solved automatically by using the local (theirs==False) or the remote (theirs==True) branch. Otherwise, or for conflicts involving the inputs of cells, the json will be repaired by including the two version of the conflicted cell(s) with markdown cells indicating the conflicts. You will be able to open the notebook again and search for the conflicts (look for <<<<<<<) then fix them as you wish.

A message will be printed indicating whether the notebook was fully merged or if conflicts remain.

nbdev_fix(broken, outname='tmp.ipynb')
chk = read_nb('tmp.ipynb')
test_eq(len(chk.cells), 7)
os.unlink('tmp.ipynb')

One or more conflict remains in the notebook, please inspect manually.

Git merge driver

---

source

nbdev_merge

 nbdev_merge (base:str, ours:str, theirs:str, path:str)

Git merge driver for notebooks

This implements a git merge driver for notebooks that automatically resolves conflicting metadata and outputs, and splits remaining conflicts as separate cells so that the notebook can be viewed and fixed in Jupyter. The easiest way to install it is by running nbdev_install_hooks.

This works by first running Git’s default merge driver, and then nbdev_fix if there are still conflicts. You can set nbdev_fix’s theirs argument using the THEIRS environment variable, for example:

THEIRS=True git merge branch