Packaging Python modules with wheels

Everyone who has been working with Python for some time may have already got come round packages. In Python terminology, packages (or distribution packages) are collections of a number of Python modules that present particular performance. The normal idea is similar to libraries in different languages. Some peculiarities with Python packages make coping with them totally different.

Pip and PyPi

The most typical method to set up a third-party Python bundle is to make use of bundle installer pip, provided by default. The Python Package Index (PyPi) is the central server for packages of every kind and the default supply for pip. Python packages include information that specify the bundle identify, model, and different meta data. Based on these information, PyPi is aware of the right way to classify and index a bundle. In addition, these information might embody set up directions that pip processes.

Source and binary distribution

Python modules are distributed in a number of codecs, every with execs and cons. In normal, the codecs may be divided into two teams.

Source distribution (sdist)

Source distributions are outlined in PEP 517 and are gzipped tar archives with the file ending *.tar.gz. The archive incorporates all package-related supply information and set up directions. A supply distribution typically has dependencies to a construct system like distutils or setuptools which trigger code execution throughout set up. The execution of (arbitrary) code upon set up might increase security considerations.

In the case of a Python C/C++ extension, a supply distribution incorporates plain C/C++ information. These should be compiled upon set up, so an applicable C/C++ toolchain should be current.

Built distributions (bdist)

In distinction, you possibly can typically use a constructed distribution as is. The thought behind constructed distributions is to offer a bundle format with out introducing extra dependencies. When it involves Python C/C++ extension, a constructed distribution offers binaries prepared for the person’s platform.

The most generally used constructed distribution format is the Python wheel, laid out in PEP 427.

Python wheels

Wheels are ZIP archives with the file ending .whl. A wheel might include binaries, scripts, or plain Python information. If a wheel incorporates binaries of a C/C++ extension module, it signifies that by together with its goal platform in its filename. Pure Python information (.py) are compiled into Python byte code (.pyc) through the set up of the wheel.

If you try to put in a bundle from PyPi utilizing pip, it all the time chooses a Python wheel over a supply distribution. However, when pip can not discover a suitable wheel, it makes an attempt to fetch the supply distribution as a substitute. As a bundle maintainer, it is a good observe to offer each codecs on pip. For a bundle person, utilizing wheels over supply distributions is advantageous due to the safer set up course of, their smaller dimension, and, in consequence, sooner set up time.

To handle a variety of customers, the bundle maintainer should provide wheels for numerous platforms and Python variations.

In considered one of my earlier articles, Write a C++ extension module for Python, I demonstrated the right way to create a Python C++ extension for the CPython interpreter. You can re-use the article’s example code to construct your first wheel.

Defining the construct configuration with setuptools

The demo repository incorporates the next information, which include meta data and an outline of the construct course of:

pyproject.toml

[build-system]
requires = [
    "setuptools>=58"
]

build-backend = "setuptools.build_meta"

This file is the successor of the setup.py since PEP 517 and PEP 518. This file is definitely the entry level for the packaging course of. The build-backend key tells pip to make use of setuptools because the construct system.

setup.cfg

This file incorporates the static, by no means altering metadata of the bundle:

[metadata]
identify = MyModule
model = 0.0.1

description = Example C/C++ extension module
long_description = Does nothing besides incremention a quantity
license = GPLv3
classifiers = 
    Operating System::Microsoft
    Operating System::POSIX::Linux
    Programming Language::C++

setup.py

This file defines the generic construct course of for the Python module. Every motion which should be carried out at set up time goes right here.

Due to safety considerations, this file ought to solely be current if completely crucial.

from setuptools import setup, Extension

MyModule = Extension(
                    'MyModule',
                    sources = ['my_py_module.cpp', 'my_class_py_type.cpp'],
                    extra_compile_args=['-std=c++17']
                    )

setup(ext_modules = [MyModule])

This instance bundle is definitely a Python C/C++ extension, so it requires a C/C++ toolchain on the person’s system to compile. In the earlier article, I used CMake to generate the construct configuration. This time, I’m utilizing setuptools for the construct course of. I confronted challenges when working CMake inside a construct container (I’ll come again to that time later). The setup.py file incorporates all the data required to construct the extension module.

In this instance, setup.py lists the concerned supply information and a few (optionally available) compile arguments. You can discover a reference to the setuptools construct within the documentation.

Build course of

To begin the construct course of, open a terminal within the root folder of the repository and run:

$ python3 -m construct --wheel

Afterward, discover the subfolder dist containing a .whl file. For instance:

MyModule-0.0.1-cp39-cp39-linux_x86_64

The file identify carries a whole lot of data. After the module identify and model, it specifies the Python interpreter (CPython 3.9) and the goal structure (x86_64).

At this level, you possibly can set up and take a look at the newly created wheel:

$ python3 -m venv venv_test_wheel/

$ supply venv_test_wheel/bin/activate

$ python3 -m pip set up dist/MyModule-0.0.1-cp39-cp39-linux_x86_64.whl

Now you could have one wheel, which you’ll ahead to somebody utilizing the identical interpreter on the identical structure. This is the naked minimal, so I’ll go one step additional and present you the right way to create wheels for different platforms.

Build configuration

As a bundle maintainer, you need to present an acceptable wheel for as many platforms as doable. Luckily, there are instruments to make this straightforward for you.

Maintaining Linux compatibility

When constructing Python C/C++ extensions, the ensuing binaries are linked towards the usual libraries of the construct system. This may trigger some incompatibilities on Linux, with its numerous variations of glibc. A Python C/C++ extension module constructed on one Linux system might not work on one other comparable Linux system because of, for instance, the dearth of a sure shared library. To avert such situations, PEP 513 proposed a tag for wheels that work on many Linux platforms: manylinux.

Building for the manylinux platform causes linking towards an outlined kernel and userspace ABI. Modules that conform to this commonplace are anticipated to work on many Linux methods. The manylinux tag developed over time, and in its newest commonplace (PEP 600), it instantly names the glibc variations the module was linked towards (manylinux_2_17_x86_64, for instance).

In addition to manylinux, there’s the musllinux platform (PEP 656), which defines a construct configuration for distributions using musl libc like Alpine Linux.

CI construct wheel

The cibuildwheel challenge offers CI construct configurations for a lot of platforms and probably the most extensively used CI/CD methods.

Many Git internet hosting platforms have CI/CD options inbuilt. The challenge is hosted on GitHub, so you should utilize GitHub Actions as a CI server. Just observe the instructions for GitHub Actions and supply a workflow file in your repository: .github/workflows/build_wheels.yml.

A push to GitHub triggers the workflow. After the workflow has completed (be aware that it took over quarter-hour to finish), you possibly can obtain an archive containing a wheel for numerous platforms:

You nonetheless need to bundle these wheels manually if you wish to publish them on PyPi. Using CI/CD, it is doable to automate the supply course of to PyPi. You can discover additional directions in cibuildwheels documentation.

Wrap up

The numerous codecs could make the packaging of Python modules an obtuse course of for freshmen. Knowledge in regards to the totally different bundle codecs, their function, and the instruments concerned within the packaging course of is critical for bundle maintainers. I hope this text sheds gentle on the world of Python packaging. In the tip, through the use of a CI/CD construct system, offering packages within the advantageous wheel format turns into a breeze.