Enhancing Python with Custom C Extensions

Introduction

This article is going to highlight the features of CPython's C API which is used to build C extensions for Python. I will be going over the general workflow for taking a small library of fairly banal, toy example, C functions and exposing it to a Python wrapper.

You might be wondering... Python is a fantastic high level language capable of just about anything, why would I want to deal with messy C code? And I would have to agree with the general premise of that argument. However, there are two common use cases I have found where this is likely to come up: (i) to speed up a particular slow piece of Python code and, (ii) you are forced to include a program already written in C into an establish Python program and you don't want to rewrite the C code in Python. The latter happened to me recently and I wanted to share what I've learned with you.

Summary of Key Steps

1. Obtain or write C code
2. Write Python C API wrapper function
3. Define function(s) table
4. Define module
5. Write initialization function
6. Package and build the extension

Obtaining or Writing C Code

For this tutorial I will be working with a small set of C functions that I wrote with my limited knowledge of C. All C programmers reading this please have pity on me for the code you are about to see.

// demolib.h
unsigned long cfactorial_sum(char num_chars[]);
unsigned long ifactorial_sum(long nums[], int size);
unsigned long factorial(long n);

#include <stdio.h>
#include "demolib.h"

unsigned long cfactorial_sum(char num_chars[]) {
    unsigned long fact_num;
    unsigned long sum = 0;

    for (int i = 0; num_chars[i]; i++) {
        int ith_num = num_chars[i] - '0';
        fact_num = factorial(ith_num);
        sum = sum + fact_num;
    }
    return sum;
}

unsigned long ifactorial_sum(long nums[], int size) {
    unsigned long fact_num;
    unsigned long sum = 0;
    for (int i = 0; i < size; i++) {
        fact_num = factorial(nums[i]);
        sum += fact_num;
    }
    return sum;
}

unsigned long factorial(long n) {
    if (n == 0)
        return 1;
    return (unsigned)n * factorial(n-1);
}

The first file demolib.h is a C header file that defines the function signatures I will be working with and the second file demolib.c shows the actual implementations of those functions.

The first function cfactorial_sum(char num_chars[]) receives a C string of numerical digits represented by an array of chars where each char is a number. The function builds a sum by looping over each char, converting it to an int, calculating the factorial of that int via factorial(long n) and adding it to the cumulative sum. Finally it returns the sum to the client code calling it.

The second function ifactorial_sum(long nums[], int size) behaves similarly to sfactorial_sum(...), but without the need to convert to ints.

The last function is a simple factorial(long n) function implemented in a recursive type algorithm.

Writing Python C API Wrapper Functions

Writing the C to Python wrapper function is the most involved part of the whole process I am going to demonstrate. The Python C extension API I will be using are in the C header file Python.h, which comes included with most CPython installations. For the purpose of this tutorial I will be using the anaconda distribution of CPython 3.6.

First things first, I will include the Python.h header file at the top of a new file called demomodule.c, and I will also include my custom header file demolib.h as it sort of serves as an interface to the functions I will be wrapping. I should also add that all the files we are working with should be in the same directory.

// demomodule.c
#include <Python.h>
#include "demolib.h"

Now I will begin working on defining the wrapper to the first C function cfactorial_sum(...). The function needs to be static as its scope should be limited only to this file and it should return a PyObject exposed to our program via the Python.h header file. The wrapper function name will be DemoLib_cFactorialSum and it will contain two arguments, both of type PyObject with the first being a pointer to self and the second a pointer to the args passed to the function via the calling Python code.

static PyObject *DemoLib_cFactorialSum(PyObject *self, PyObject *args) {
    ...
}

Next I need to parse the string of digits that the client Python code will be passing to this function and convert it to a C chars array so it can be used by the cfactorial_sum(...) function to return the factorial sum. I will do this using PyArg_ParseTuple(...).

First I will need to define a C char pointer called char_nums which will receive the contents of the Python string being passed to the function. Next I'll call PyArg_ParseTuple(...) passing it the PyObject args value, a format string "s" that specifies that the first (and only) parameter of args is a string which should be coerced into the last argument, the char_nums variable.

If an error occurs in PyArg_ParseTuple(...) it will raise the appropriate type error exception and the return value will be zero, which is interpreted as false in a conditional. If an error is detected in my if-statement I return a NULL, which signals to the calling Python code that an exception occurred.

static PyObject *DemoLib_cFactorialSum(PyObject *self, PyObject *args) {
    char *char_nums;
    if (!PyArg_ParseTuple(args, "s", &char_nums)) {
        return NULL:
    }
}

I would like to take a little time to talk about how the PyArg_ParseTuple(...) function works. I have built a mental model around the function such that I see it as taking the variable number of positional arguments passed to the client Python function and captured by the PyObject *args parameter. I then think of the arguments captured by the *args parameter as being unpacked into the C-defined variables that come after the format string specifier.

The below table shows what I feel are the more commonly used format specifiers.

Specifier	C Type	Description
c	char	Python string of length 1 converted to C char
s	char array	Python string converted to C char array
d	double	Python float converted to a C double
f	float	Python float converted to a C float
i	int	Python int converted to a C int
l	long	Python int converted to a C long
o	PyObject *	Python object converted to a C PyObject

If you are passing multiple arguments to a function which are to be unpacked and coerced into C types, then you simply use multiple specifiers such as PyArg_ParseTuple(args, "si", &charVar, &intVar).

Ok, now that we've got a feel for how PyArg_ParseTuple(...) works I will move along. The next thing to do is to call the cfactorial_sum(...) function passing it the char_nums array that we just built from the Python string that was passed to the wrapper. The return will be an unsigned long.

static PyObject *DemoLib_cFactorialSum(PyObject *self, PyObject *args) {
    // arg parsing omitted
    unsigned long fact_sum;
    fact_sum = cfactorial_sum(char_nums);
}

The last thing to do in the DemoLib_cFactorialSum(...) wrapper function is to return the sum in a form that the client Python code can work with. To do this I use another tool called Py_BuildValue(...) exposed via the Python.h treasure trove. Py_BuildValue uses format specifiers very similar to how PyArg_ParseTuple(...) uses them, just in the opposite direction. Py_BuildValue also allows for returning our familiar Python data structures such as tuples and dicts. In this wrapper function I will be returning an int to Python, which I implement as follows:

static PyObject *DemoLib_cFactorialSum(PyObject *self, PyObject *args) {
    // arg parsing omitted

    // factorial function call omitted

    return Py_BuildValue("i", fact_sum);
}

Here are some examples of some of the other return value formats and types:

Wrapper Code	Returned to Python
Py_BuildValue("s", "A")	"A"
Py_BuildValue("i", 10)	10
Py_BuildValue("(iii)", 1, 2, 3)	(1, 2, 3)
Py_BuildValue("{si,si}", "a', 4, "b", 9)	{"a": 4, "b": 9}
Py_BuildValue("")	None

Cool, right!?

Now let's get on to implementing the wrapper to the other C function ifactorial_sum(...). This wrapper is going to include a few other quirks to work through.

static PyObject *DemoLib_iFactorialSum(PyObject *self, PyObject *args) {
    PyObject *lst;
    if(!PyArg_ParseTuple(args, "O", &lst)) {
        return NULL;
    }
}

As you can see, the function signature is the same as the last example in that it is static, returns a PyObject, and the parameters are two PyObjects. However, the argument parsing is a little different. Since the Python function will be passed a list which does not have a recognizable C type I need to utilize more tooling of the Python C API. The "O" format specifier in PyArg_ParseTuple indicates that a PyObject is expected, which gets assigned to the generic PyObject *lst variable.

Behind the scenes the Python C API machinery recognizes that the argument passed in implements the sequence interface, which allows me to get the size of the passed in list using the PyObject_Length function. If this function is given a PyObject type that does not implement the sequence interface then a NULL is returned.

    int n = PyObject_Length(lst);
    if (n < 0) {
        return NULL;
    }

Now that I know the size of the list I can convert its elements to a C array of ints and feed that into my ifactorial_sum C function that was defined previously. To do this I use a for-loop to iterate over the elements of the list, retrieving each item using PyList_GetItem, which returns a PyObject implemented as a Python representation of a long called PyLongObject. I then use PyLong_AsLong to convert the Python representation of a long into the common C long data type and populate the C array of longs I've named nums.

  long nums[n];
  for (int i = 0; i < n; i++) {
    PyLongObject *item = PyList_GetItem(lst, i);
    long num = PyLong_AsLong(item);
    nums[i] = num;
  }

At this point I can call my ifactorial_sum(...) function passing it nums and n, which returns the factorial sum of the array of longs. Again, I'll use Py_BuildValue to convert the sum back into a Python int and return it to the calling client Python code.

    unsigned long fact_sum;
    fact_sum = ifactorial_sum(nums, n);

    return Py_BuildValue("i", fact_sum);

The remainder of the code to be written is simply boilerplate Python C API code which I will spend less time explaining and refer the reader to the docs for details.

Define Function(s) Table

In this section I will write out an array which associates the two wrapper functions written in the previous section to the name which will be exposed in Python. This array also indicates the type of arguments which are passed to our functions, METH_VARARGS, and provides a function-level doc string.

static PyMethodDef DemoLib_FunctionsTable[] = {
    {
        "sfactorial_sum",      // name exposed to Python
        DemoLib_cFactorialSum, // C wrapper function
        METH_VARARGS,          // received variable args (but really just 1)
        "Calculates factorial sum from digits in string of numbers" // documentation
    }, {
        "ifactorial_sum",      // name exposed to Python
        DemoLib_iFactorialSum, // C wrapper function
        METH_VARARGS,          // received variable args (but really just 1)
        "Calculates factorial sum from list of ints" // documentation
    }, {
        NULL, NULL, 0, NULL
    }
};

Define Module

Here I will provide a module definition which associates the previously defined DemoLib_FunctionsTable array to the module. This struct is also responsible for defining the name of the module that is exposed in Python as well as giving a module-level doc string.

static struct PyModuleDef DemoLib_Module = {
    PyModuleDef_HEAD_INIT,
    "demo",     // name of module exposed to Python
    "Demo Python wrapper for custom C extension library.", // module documentation
    -1,
    DemoLib_FunctionsTable
};

Write the Initialization Function

The last C-ish bit of code to write is the module's initialization function, which is the only non-static member of the wrapper code. This function has a very particular naming convention of PyInit_name where name is the name of the module. This function is invoked in the Python interpreter, which creates the module and makes it accessible.

PyMODINIT_FUNC PyInit_demo(void) {
    return PyModule_Create(&DemoLib_Module);
}

The complete extension code now looks like this:

#include <stdio.h>
#include <Python.h>
#include "demolib.h"

// wrapper function for cfactorial_sum
static PyObject *DemoLib_cFactorialSum(PyObject *self, PyObject *args) {
    char *char_nums;
    if (!PyArg_ParseTuple(args, "s", &char_nums)) {
        return NULL;
    }

    unsigned long fact_sum;
    fact_sum = cfactorial_sum(char_nums);

    return Py_BuildValue("i", fact_sum);
}

// wrapper function for ifactorial_sum
static PyObject *DemoLib_iFactorialSum(PyObject *self, PyObject *args) {
    PyObject *lst;
    if (!PyArg_ParseTuple(args, "O", &lst)) {
        return NULL;
    }

    int n = PyObject_Length(lst);
    if (n < 0) {
        return NULL;
    }

    long nums[n];
    for (int i = 0; i < n; i++) {
        PyLongObject *item = PyList_GetItem(lst, i);
        long num = PyLong_AsLong(item);
        nums[i] = num;
    }

    unsigned long fact_sum;
    fact_sum = ifactorial_sum(nums, n);

    return Py_BuildValue("i", fact_sum);
}

// module's function table
static PyMethodDef DemoLib_FunctionsTable[] = {
    {
        "sfactorial_sum", // name exposed to Python
        DemoLib_cFactorialSum, // C wrapper function
        METH_VARARGS, // received variable args (but really just 1)
        "Calculates factorial sum from digits in string of numbers" // documentation
    }, {
        "ifactorial_sum", // name exposed to Python
        DemoLib_iFactorialSum, // C wrapper function
        METH_VARARGS, // received variable args (but really just 1)
        "Calculates factorial sum from list of ints" // documentation
    }, {
        NULL, NULL, 0, NULL
    }
};

// modules definition
static struct PyModuleDef DemoLib_Module = {
    PyModuleDef_HEAD_INIT,
    "demo",     // name of module exposed to Python
    "Demo Python wrapper for custom C extension library.", // module documentation
    -1,
    DemoLib_FunctionsTable
};

PyMODINIT_FUNC PyInit_demo(void) {
    return PyModule_Create(&DemoLib_Module);
}

Packaging and Building the Extension

Now I will package and build the extension so I can use it in Python with the help of the Setuptools library.

The first thing I will need to do is install setuptools:

$ pip install setuptools

Now I'll create a new file called setup.py. Below is a representation of how my files are organized:

├── demolib.c
├── demolib.h
├── demomodule.c
└── setup.py

Inside setup.py place the following code, which imports the Extension class and the setup function from setuptools. I instantiate the Extension class which is used to compile the C code using the GCC compiler, which is natively installed on most Unix-style operating systems. Windows users will want to install MinGW.

The last bit of code shown simply passes in the minimal suggested information to package the code into a Python package.

from setuptools import Extension, setup

module = Extension("demo",
                  sources=[
                    'demolib.c',
                    'demomodule.c'
                  ])
setup(name='demo',
     version='1.0',
     description='Python wrapper for custom C extension',
     ext_modules=[module])

In a shell I will execute the following command to build and install the package onto my system. This code will locate the setup.py file and call its setup(...) function:

$ pip install .

Finally, I can now fire up a Python interpreter, import my module and test my extension functions:

$  python
Python 3.6.4 |Anaconda, Inc.| (default, Dec 21 2017, 15:39:08)
>>> import demo
>>> demo.sfactorial_sum("12345")
153
>>> demo.ifactorial_sum([1,2,3,4,5])
153
>>>

Conclusion

In my concluding remarks I would like to say that this tutorial really barely scratches the surface of the Python C API, which I found to be a huge and daunting topic. It is my hope that should you find yourself needing to extend Python, this tutorial along with the official docs aid you in achieving that goal.

Thanks for reading and I welcome any and all comments or criticisms below.