Debugging inside jupyter notebooks

The functionality we are about to explore here has quickly become my all time favorite.

These commands are not only great for debugging but also for hands on experimentation and learning.

Just a cautionary note - you will not be able to do 'Cell > run all'. Also, if you don't properly exit out from the debugger, you might need to restart the kernel.

%debug magic

How to use:

  1. Get an exception.
  2. Insert a new cell, type %debug and run it.
In [1]:
def full_speed_ahead(engine_power=50):
    if engine_power > 100: raise ValueError(
        'Oh no, you set the engine_power to above 100! Things will start to fall apart!'
    )
In [2]:
full_speed_ahead(125)
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-2-3241c44eda9c> in <module>()
----> 1 full_speed_ahead(125)

<ipython-input-1-a180baaf6aad> in full_speed_ahead(engine_power)
      1 def full_speed_ahead(engine_power=50):
      2     if engine_power > 100: raise ValueError(
----> 3         'Oh no, you set the engine_power to above 100! Things will start to fall apart!'
      4     )

ValueError: Oh no, you set the engine_power to above 100! Things will start to fall apart!
In [3]:
%debug
> <ipython-input-1-a180baaf6aad>(3)full_speed_ahead()
      1 def full_speed_ahead(engine_power=50):
      2     if engine_power > 100: raise ValueError(
----> 3         'Oh no, you set the engine_power to above 100! Things will start to fall apart!'
      4     )

ipdb> engine_power
125
ipdb> u
> <ipython-input-2-3241c44eda9c>(1)<module>()
----> 1 full_speed_ahead(125)

ipdb> d
> <ipython-input-1-a180baaf6aad>(3)full_speed_ahead()
      1 def full_speed_ahead(engine_power=50):
      2     if engine_power > 100: raise ValueError(
----> 3         'Oh no, you set the engine_power to above 100! Things will start to fall apart!'
      4     )

ipdb> exit

The %debug magic opens an interactive debugger and takes you to where the exception occured.

You can take a look around and interact with variables that are in scope. You can also navigate up and down the stack using the up and down commands.

To access a list of available commands type help or h (its enough to enter only the first letter).

To exit the debugger, execute exit or just e.

Setting arbitrary break points

In [4]:
from IPython.core.debugger import set_trace

You can now open an interactive debugger from a point of your choosing in your code!

All you have to do is add the set_trace() call.

Important: you can also add this to code that lives outside of your notebook. Say a utils.py module you are importing.

In [5]:
def full_speed_ahead_without_breaks(engine_power=50):
    if engine_power > 100: set_trace() # let's see what happens if we go that fast
In [6]:
full_speed_ahead_without_breaks()
In [7]:
full_speed_ahead_without_breaks(125)
--Return--
None
> <ipython-input-5-c6fa41d50140>(2)full_speed_ahead_without_breaks()
      1 def full_speed_ahead_without_breaks(engine_power=50):
----> 2     if engine_power > 100: set_trace() # let's see what happens if we go that fast

ipdb> exit
---------------------------------------------------------------------------
BdbQuit                                   Traceback (most recent call last)
<ipython-input-7-a261f420f8c8> in <module>()
----> 1 full_speed_ahead_without_breaks(125)

<ipython-input-5-c6fa41d50140> in full_speed_ahead_without_breaks(engine_power)
      1 def full_speed_ahead_without_breaks(engine_power=50):
----> 2     if engine_power > 100: set_trace() # let's see what happens if we go that fast

~/anaconda3/lib/python3.6/bdb.py in trace_dispatch(self, frame, event, arg)
     50             return self.dispatch_call(frame, arg)
     51         if event == 'return':
---> 52             return self.dispatch_return(frame, arg)
     53         if event == 'exception':
     54             return self.dispatch_exception(frame, arg)

~/anaconda3/lib/python3.6/bdb.py in dispatch_return(self, frame, arg)
     94             finally:
     95                 self.frame_returning = None
---> 96             if self.quitting: raise BdbQuit
     97             # The user issued a 'next' or 'until' command.
     98             if self.stopframe is frame and self.stoplineno != -1:

BdbQuit: 

Hands-on learning

It's easier to figure out how to use something if you can experiment with it.

But how can we do so with things hidden multiple layers below the the API?

For instance, can we take a look at what functions that we pass to other functions receive?

In [8]:
d = [(1, 2), (3, 4)]
sorted(d, key=lambda itm: set_trace())
--Return--
None
> <ipython-input-8-9879382add4b>(2)<lambda>()
      1 d = [(1, 2), (3, 4)]
----> 2 sorted(d, key=lambda itm: set_trace())

ipdb> itm
(1, 2)
ipdb> exit
---------------------------------------------------------------------------
BdbQuit                                   Traceback (most recent call last)
<ipython-input-8-9879382add4b> in <module>()
      1 d = [(1, 2), (3, 4)]
----> 2 sorted(d, key=lambda itm: set_trace())

<ipython-input-8-9879382add4b> in <lambda>(itm)
      1 d = [(1, 2), (3, 4)]
----> 2 sorted(d, key=lambda itm: set_trace())

~/anaconda3/lib/python3.6/bdb.py in trace_dispatch(self, frame, event, arg)
     50             return self.dispatch_call(frame, arg)
     51         if event == 'return':
---> 52             return self.dispatch_return(frame, arg)
     53         if event == 'exception':
     54             return self.dispatch_exception(frame, arg)

~/anaconda3/lib/python3.6/bdb.py in dispatch_return(self, frame, arg)
     94             finally:
     95                 self.frame_returning = None
---> 96             if self.quitting: raise BdbQuit
     97             # The user issued a 'next' or 'until' command.
     98             if self.stopframe is frame and self.stoplineno != -1:

BdbQuit: 

But maybe that is too trivial. Let's try a more real world scenario.

In [9]:
import pandas as pd
In [10]:
df = pd.DataFrame(data=pd.np.random.randn(4, 2), columns=['a', 'b'])

What does the function passed to apply receive?

In [11]:
df
Out[11]:
a b
0 -0.195081 -1.159500
1 0.504801 0.926347
2 0.427790 -1.338200
3 0.124840 -0.575361
In [12]:
df.apply(lambda smth: set_trace())
--Return--
None
> <ipython-input-12-6072fd4e284c>(1)<lambda>()
----> 1 df.apply(lambda smth: set_trace())

ipdb> smth
0   -0.195081
1    0.504801
2    0.427790
3    0.124840
Name: a, dtype: float64
ipdb> exit
--Return--
None
> <ipython-input-12-6072fd4e284c>(1)<lambda>()
----> 1 df.apply(lambda smth: set_trace())

ipdb> exit
---------------------------------------------------------------------------
BdbQuit                                   Traceback (most recent call last)
<ipython-input-12-6072fd4e284c> in <module>()
----> 1 df.apply(lambda smth: set_trace())

~/anaconda3/lib/python3.6/site-packages/pandas/core/frame.py in apply(self, func, axis, broadcast, raw, reduce, args, **kwds)
   4875                         f, axis,
   4876                         reduce=reduce,
-> 4877                         ignore_failures=ignore_failures)
   4878             else:
   4879                 return self._apply_broadcast(f, axis)

~/anaconda3/lib/python3.6/site-packages/pandas/core/frame.py in _apply_standard(self, func, axis, ignore_failures, reduce)
   4931                     labels = self._get_agg_axis(axis)
   4932                     result = lib.reduce(values, func, axis=axis, dummy=dummy,
-> 4933                                         labels=labels)
   4934                     return Series(result, index=labels)
   4935                 except Exception:

<ipython-input-12-6072fd4e284c> in <lambda>(smth)
----> 1 df.apply(lambda smth: set_trace())

~/anaconda3/lib/python3.6/bdb.py in trace_dispatch(self, frame, event, arg)
     50             return self.dispatch_call(frame, arg)
     51         if event == 'return':
---> 52             return self.dispatch_return(frame, arg)
     53         if event == 'exception':
     54             return self.dispatch_exception(frame, arg)

~/anaconda3/lib/python3.6/bdb.py in dispatch_return(self, frame, arg)
     94             finally:
     95                 self.frame_returning = None
---> 96             if self.quitting: raise BdbQuit
     97             # The user issued a 'next' or 'until' command.
     98             if self.stopframe is frame and self.stoplineno != -1:

BdbQuit: occurred at index a