Cython notes and tips
This post is a collection of notes and tips about Cython, that I learned or discovered while developping for scikit-learn. I will describe my (basic) workflow, and I will describe what I learned about avoiding Python interactions, memory views, the GIL and other fun stuff.
This is neither a tutorial, nor an introduction to Cython. This is rather a list of things I wish I knew before writing a few thousands lines of Cython code. I hope it can be useful to others!
My Cython workflow is pretty basic. Write code, compile, fix compilation errors, then run the tests… Nothing new here. There are a few Cython-specific steps though. Since Cython is pretty magical, a small change in your code can induce significant drops (or gains) in performance. I strongly suggest to:
- Always benchmark your code, extensively.
- Always track down Python interactions. Yellow is your enemy (see below).
About debugging: there is, AFAIK, no debugger for Cython. You can of course
still use a debugger for the generated C-code, but that’s not really
convenient. Personally, I use the good old
print() statements with
extensive unit tests. Tip from Jérémie in the comments: inside a
print() statements are forbidden), you can temporarily
re-acquire the gil back with
with gil: and use
Python interactions and how to avoid them
Cython generates C code that conceptually operates in 2 different modes: either in “Python mode” or in “pure C mode”.
Python mode is when the code manipulates Python objects, through the Python/C API: for example when you are using a dict, or a numpy array. A call to this Python/C API is called a Python interaction.
Pure C mode is when the code only manipulates pure C types (things that are
cdef‘ed) and does not make any use of the Python/C API. For example if
you want to manipulate a numpy array in pure C mode, use a memory view
instead (see below).
My simplistic proxy is that Python interaction = slow = bad, while pure C
mode = fast = good. When writing Cython, you want to avoid Python
interactions as much as possible, especially deep inside
for loops. The
cython -a file.pyx will output a html file of the generated C
code, where each line has a different shade of yellow: more yellow means
more Python interactions.
When I write Cython code, I always use
cython -a on every file. My goal
is that all the file is white (i.e. no interaction), although interactions
are unavoidable in some places: typically when arguments are passed in, or
when returning Python objects.
Some tips to avoid Python interactions:
cdefas much as you can
- Use memory views to manipulate numpy arrays. If your function accepts a numpy
array, declare a
cdef‘ed memory view to manipulate it.
- A good way to make sure that you’re not risking any interaction is to write
code within a
with nogilcontext manager (see below).
- For some reason, declaring a variable locally in a function may make some
interactions disappear. Typically, I noticed that using a local variable as
an alias to the attribute of a
cdef‘ed class will remove Python interactions (
cdef int local_var = self.var). Don’t follow these tips blindly though. In some cases it will make your code faster, but sometimes it won’t. In the end, always let your benchmarks decide.
Multi-threading and the GIL
The GIL is this
annoying thing that prevents programs run with
CPython to do
multi-threaded parallelism. You can do multi-processing (e.g. with
joblib), but unless the GIL is
“released”, multi-threading isn’t possible.
Cython allows you to release the GIL. That means that you can do multi-threading in at least 2 ways:
- Directly in Cython, using OpenMP with prange.
- Using e.g.
joblibwith a multi-threading backend (the parts of your code that will be parallelized are the parts that release the GIL)
We use both in scikit-learn.
To release the GIL in Cython, you just need to use the
context manager (you can also do that directly when using
Your code inside of a
nogil statement cannot have any Python interaction.
Any variable you use will have to be
cdef‘ed, and you won’t be allowed to
use numpy arrays since these are objects: use views instead! If you call a
function, it needs to be labeled as a
nogil function, like so:
cdef void my_func() nogil: # ...
This is carefully explained in the docs, but I didn’t really get it at
first: labelling a function as
nogil does not release the GIL. It
simply tells Cython that the function may be called without the GIL
(Cython uses this information to do some static compilation-time checks).
As a result this function must not have any kind of Python interaction.
You are still allowed to call the function with the GIL, though.
Another thing that wasn’t clear for me at first: it is perfectly OK to
release the GIL inside of a
def function that has some Python
interactions, as long as the Python interactions happen outside of the
nogil block. In other words, no need to overthink your functions
interfaces. For example, this is perfectly possible:
def f(array): # Take Python object as input cdef: int [:] my_view = array # Python interactions possible here # ... with nogil: # No Python interactions here # ... # do stuff to my_view, maybe in parallel with prange... # ... # Python interactions possible here # ... return array # return a Python object # Then from Python, in a .py file: out = f(np.arange(10))
Finally: releasing and aquiring the GIL takes time. You don’t want to do that deep inside nested for loops. It is instead better to write big chunks of nogil code.
Using numpy arrays and memory views
In scikit-learn we rely on numpy arrays for almost everything. Cython supports numpy arrays but since these are Python objects, we can’t manipulate them without the GIL.
In the past, the workaround was to use pointers on the data, but that can get ugly very quickly, especially when you need to care about the memory alignment of 2D arrays (C vs Fortran).
Cython now supports memory views, which can be used without the GIL. A view is a light struct that basically contains a pointer to the raw data, and info about the type, memory alignment, etc.
The interaction between numpy arrays and views is pretty flexible. In particular, the following patterns are perfectly acceptable:
# Declare an argument as a view, and pass in an array def f(int [:] my_view): # ... f(np.arange(10))
# Take an array as input, and locally map it to a view def f(array): cdef int [:] my_view = array # bind array to a view # You can now manipulate the view, possibly without the GIL
# Allocate an array inside of a function, and manipulate it with a view. def f(): cdef int [:] my_view = np.arange(10) # You can now manipulate the view, possibly without the GIL return np.asarray(my_view)
That might seem obvious, but views do not support the numpy array API: You
cannot do things like
my_view.mean(). You can do
np.mean(my_view), but not
without the GIL.
I find myself not using pointers at all, unless I really need to
free() something in a
Other tips about memory views:
You can manipulate views over numpy structured array (i.e. arrays with a complex dtype): see this PR, which isn’t merged at the time of writing.
Since you’re writing C-like code, in general you’ll want to disable the bounds checks and the wraparound to make your code faster (see the docs).
If you know whether your array is C-aligned or Fortran-aligned, definitely let Cython know. Else, Cython will generate general code that can work with arbitrary alignment, which is less optimized:
cdef int [:, ::1] my_view # C aligned (contiguous on the last dim) cdef int [::1, :] my_view # Fortran aligned (contiguous on the 1st dim)
Cython is an amazing tool, and the whole Python data-science ecosystem owes Cython a lot. Scikit-learn, Scipy and pandas heavily rely on it.
Much like Numba, it can however be a bit (too) magical sometimes, and even the smallest change can have a huge impact on the performance of your code, sometimes for obscure reasons. I can’t stress this enough: always benchmark your code.
The Cython documentation is full of great tips, though with time the organization is becoming a bit confusing (with e.g. some redundancy between the User Guide and the tutorials). This Scipy paper also has some useful info that are not all covered in the docs.
I also found the Cython book by Kurt W. Smith to be very clear and useful. It covers most of the topics from the docs in a very accessible way.