Write a C++ extension module for Python

In a earlier article, I gave an outline of six Python interpreters. On most techniques, the CPython interpreter is the default, and in addition the ballot in my final article confirmed that CPython is the most well-liked one. Specific to CPython is the power to jot down Python modules in C utilizing CPythons extensions API. Writing Python modules in C means that you can transfer computation-intensive code to C whereas preserving the convenience of entry of Python.

In this text, I’ll present you methods to write an extension module. Instead of plain C, I exploit C++ as a result of most compilers normally perceive each. I’ve to say one main downside upfront: Python modules constructed this manner usually are not transportable to different interpreters. They solely work at the side of the CPython interpreter. So if you’re on the lookout for a extra transportable means of interacting with C libraries, think about using the ctypes module.

Source code

As typical, you will discover the associated supply code on GitHub. The C++ recordsdata within the repository have the next goal:

• my_py_module.cpp: Definition of the Python module MyModule
• my_cpp_class.h: A header-only C++ class which will get uncovered to Python
• my_class_py_type.h/cpp: Python illustration of our C++ class
• pydbg.cpp: Separate software for debugging goal

The Python module you construct on this article gained’t have any significant use, however is an efficient instance.

Build the module

Before wanting into the supply code, you’ll be able to examine whether or not the module compiles in your system. I use CMake for creating the construct configuration, so CMake should be put in in your system. In order to configure and construct the module, you’ll be able to both let Python run the method:

$ python3 setup.py construct

Or run the method manually:

$ cmake -B construct
$ cmake --build construct

After that, you have got a file known as MyModule.so within the /construct subdirectory.

Defining an extension module

First, have a look on my_py_module.cpp, specifically, the perform PyInit_MyModule:

PyMODINIT_FUNC
PyInit_MyModule(void) {
    PyObject* module = PyModule_Create(&my_module);
    
    PyObject *myclass = PyType_FromSpec(&spec_myclass);
    if (myclass == NULL){
        return NULL;
    }
    Py_INCREF(myclass);
    
    if(PyModule_AddObject(module, "MyClass", myclass) < 0){
        Py_DECREF(myclass);
        Py_DECREF(module);
        return NULL;
    }
    return module;
}

This is crucial code on this instance as a result of it acts because the entry level for CPython. In basic, when a Python C extension is compiled and made accessible as a shared object binary, CPython searches for the perform PyInit_<ModuleName> within the eponymous binary (<ModuleName>.so) and executes it when trying to import it.

All Python sorts, whether or not declarations or situations, are uncovered as tips that could PyObject. In the primary a part of this perform, the foundation definition of the module is created by working PyModule_Create(...). As you’ll be able to see within the module specification (my_module, similar file), it doesn’t have any particular performance.

Afterward, PyType_FromSpec known as to create a Python heap type definition for the customized sort MyClass. A heap sort corresponds to a Python class. The sort definition is then assigned to the module MyModule.

Note that if one of many capabilities fails, the reference depend of beforehand created PyObjects should be decremented in order that they get deleted by the interpreter.

Specifying a Python sort

The specification for the sort MyClass is discovered inside my_class_py_type.h for instance of PyType_Spec:

static PyType_Spec spec_myclass =  Py_TPFLAGS_BASETYPE,   // flags
    MyClass_slots                               // slots
;

This construction defines some primary sort info for the category. The worth handed for the scale consists of the scale of the Python illustration (MyClassObject) and the scale of the plain C++ class (MyClass). The MyClassObject is outlined as follows:

typedef struct {
    PyObject_HEAD
    int         m_value;
    MyClass*    m_myclass;
} MyClassObject;

The Python illustration is principally of sort PyObject, outlined by the macro PyObject_HEAD, and a few extra members. The member m_value is uncovered as atypical class member whereas the member m_myclass is just accessible from inside C++ code.

The PyType_Slot defines some extra performance:

static PyType_Slot MyClass_slots[] = {
    {Py_tp_new,     (void*)MyClass_new},
    {Py_tp_init,    (void*)MyClass_init},
    {Py_tp_dealloc, (void*)MyClass_Dealloc},
    {Py_tp_members, MyClass_members},
    {Py_tp_methods, MyClass_methods},
    {0, 0} /* Sentinel */
};

Here, the bounce addressed for some initialization and de-initialization capabilities are set in addition to atypical class strategies and members. Additional performance, like assigning an preliminary attribute dictionary, may be set, however that is non-compulsory. Those definitions normally finish with a sentinel, consisting of NULL values.

To full the sort specification, right here is the tactic and member desk:

static PyMethodDef MyClass_methods[] = {
    {"addOne", (PyCFunction)MyClass_addOne, METH_NOARGS,  PyDoc_STR("Return an incrmented integer")},
    {NULL, NULL} /* Sentinel */
};

static struct PyMemberDef MyClass_members[] = {
    {"value", T_INT, offsetof(MyClassObject, m_value)},
    {NULL} /* Sentinel */
};

In the tactic desk, the Python technique addOne is outlined, and it factors to the associated perform MyClass_addOne. This perform acts as a wrapper. It invokes the addOne() technique within the C++ class.

In the member desk, there is only one member outlined for demonstration functions. Unfortunately, the usage of offsetof in PyMemberDef doesn’t permit C++ particular sorts to be added to MyClassObject. If you attempt to place some C++ sort container (equivalent to std::optional), the compiler complains about it within the type of warnings associated to reminiscence format.

Initialization and de-initialization

The technique MyClass_new acts solely to supply preliminary values for MyClassObject and allocates reminiscence for the bottom sort:

PyObject *MyClass_new(PyTypeObject *sort, PyObject *args, PyObject *kwds){
    std::cout << "MtClass_new() called!" << std::endl;

    MyClassObject *self;
    self = (MyClassObject*) type->tp_alloc(sort, 0);
    if(self != NULL){ // -> allocation successfull
        // assign preliminary values
        self->m_value   = 0;
        self->m_myclass = NULL; 
    }
    return (PyObject*) self;
}

Actual initialization takes place in MyClass_init, which corresponds to the __init__() technique in Python:

int MyClass_init(PyObject *self, PyObject *args, PyObject *kwds){
    
    ((MyClassObject *)self)->m_value = 123;
    
    MyClassObject* m = (MyClassObject*)self;
    m->m_myclass = (MyClass*)PyObject_Malloc(sizeof(MyClass));

    if(!m->m_myclass){
        PyErr_SetString(PyExc_RuntimeError, "Memory allocation failed");
        return -1;
    }

    attempt {
        new (m->m_myclass) MyClass();
    } catch (const std::exception& ex) {
        PyObject_Free(m->m_myclass);
        m->m_myclass = NULL;
        m->m_value   = 0;
        PyErr_SetString(PyExc_RuntimeError, ex.what());
        return -1;
    } catch(...) {
        PyObject_Free(m->m_myclass);
        m->m_myclass = NULL;
        m->m_value   = 0;
        PyErr_SetString(PyExc_RuntimeError, "Initialization failed");
        return -1;
    }

    return 0;
}

If you need to have arguments handed throughout initialization, you will need to name PyArg_ParseTuple at this level. For the sake of simplicity, all arguments handed throughout initialization are ignored on this instance. In the primary a part of the perform, the PyObject pointer (self) is reinterpreted to a pointer to MyClassObject in an effort to get entry to our extra members. Additionally, the reminiscence for the C++ class is allotted and its constructor is executed.

Note that exception dealing with and reminiscence allocation (and de-allocation) should be rigorously carried out in an effort to forestall reminiscence leaks. When the reference depend drops to zero, the MyClass_dealloc perform takes care of releasing all associated heap reminiscence. There’s a dedicated chapter within the documentation about reminiscence administration for C and C++ extensions.

Method wrapper

Calling a associated C++ class technique from the Python class is simple:

PyObject* MyClass_addOne(PyObject *self, PyObject *args){
    assert(self);

    MyClassObject* _self = reinterpret_cast<MyClassObject*>(self);
    unsigned lengthy val = _self->m_myclass->addOne();
    return PyLong_FromUnsignedLong(val);
}

Again, the PyObject* argument (self) is casted to MyClassObject* in an effort to get entry to m_myclass, a pointer to the C++ class occasion. With this info, the courses technique addOne() known as and the result’s returned in type of a Python integer object.

3 methods to debug

For debugging functions, it may be invaluable to compile the CPython interpreter in debugging configuration. An in depth description might be discovered within the official documentation. It’s attainable to observe the following steps, so long as extra debug symbols for the pre-installed interpreter are downloaded.

GNU Debugger

Good previous GNU Debugger (GDB) is, after all, additionally helpful right here. I embody a gdbinit file, defining some choices and breakpoints. There’s additionally the gdb.sh script, which creates a debug construct and initiates a GDB session:

GDB invokes the CPython interpreter with the script file main.py. The script file means that you can simply outline all of the actions you need to carry out with the Python extension module.

C++ software

Another method is to embed the CPython interpreter in a separate C++ software. In the repository, this may be discovered within the file pydbg.cpp:

int principal(int argc, char *argv[], char *envp[])
{
    Py_SetProgramName(L"DbgPythonCppExtension");
    Py_Initialize();

    PyObject *pmodule = PyImport_ImportModule("MyModule");
    if (!pmodule) {
        PyErr_Print();
        std::cerr << "Failed to import module MyModule" << std::endl;
        return -1;
    }

    PyObject *myClassType = PyObject_GetAttrString(pmodule, "MyClass");
    if (!myClassType) {
        std::cerr << "Unable to get type MyClass from MyModule" << std::endl;
        return -1;
    }

    PyObject *myClassInstance = PyObject_CallObject(myClassType, NULL);

    if (!myClassInstance) {
        std::cerr << "Instantioation of MyClass failed" << std::endl;
        return -1;
    }

    Py_DecRef(myClassInstance); // invoke deallocation
    return 0;
}

Using the high level interface, it’s attainable to incorporate the extension module and carry out actions on it. This means that you can debug within the native IDE setting. It additionally offers you finer management of the variables handed from and to the extension module.

The downside is the excessive expense of making an additional software.

VSCode and VSCodium LLDB extension

Using a debugger extension like CodeLLDB might be essentially the most handy debugging possibility. The repository contains the VSCode or VSCodium configuration recordsdata for constructing the extension (task.json, CMake Tools) and invoking the debugger (launch.json). This method combines some great benefits of the earlier ones: Debugging in an graphical IDE, defining actions in a Python script file and even dynamically within the interpreter immediate.

Extend C++ with Python

All performance accessible from Python code can also be accessible from inside a C or C++ extension. While coding in Python is commonly thought of as a straightforward win, extending Python in C or C++ will also be a ache. On the opposite hand, whereas native Python code is slower than C++, a C or C++ extension makes it attainable to raise a computation-intensive activity to the pace of native machine code.

You should additionally think about the utilization of an ABI. The steady ABI offers a method to keep backwards compatibility to older variations of CPython as described in the documentation.

In the top, you will need to weigh the benefits and drawbacks your self. Should you determine to make use of C extensions to make sure performance accessible to you in Python, you’ve seen how it may be carried out.