Call a C API from Python becomes more enjoyable with CFFI


Jean-Sébastien Bevilacqua


What a journey !

/me

twitter/@realitix

github.com/realitix

realitix.github.io

Why Python extension ?

Accessing low-level API
OpenGL / Vulkan

Linking to existing C library

Improving performance

Our good friend is always here !

I'm totally lost
with
CPython API

Application programming interface

Application binary interface

Why ABI ?

Cython

Incremental optimization
py++

def fib(int n):
    cdef int a = 0
    cdef int b =  1
    while b < n:
        print(b)
        a, b = b, a + b

ABI / API

ctypes

Built-in / ABI ONLY

from ctypes import cdll, Structure, c_int, c_double

lib = cdll.LoadLibrary('./vector.so')

# ctypes Point structure
class Point(Structure):
    _fields_ = [('x', c_int), ('y', c_int)]

# Initialize Point[2] argument
points_array = (Point * 2)((1, 2), (3, 4))

# Get vector_size from library
vector_size_fn = lib.vector_size
vector_size_fn.restype = c_double

# Call vector_size with ctypes
size = vector_size_fn(points_array)
print('out = {}'.format(size))

cffi enlightened

in-line : import time

out-of-line : install time

ABI / In-line

from cffi import FFI
ffi = FFI()

ffi.cdef("int printf(const char *format, ...);")

C = ffi.dlopen(None)

arg = ffi.new("char[]", b"amazing")
C.printf("cffi is %s\n", arg)

$ python demo1.py
cffi is amazing

API / out-of-line

Step 1 - Builder source

from cffi import FFI

ffibuilder = FFI()

ffibuilder.set_source("_demo2",
   r"""#include <sys/types.h>
       #include <pwd.h>
    """)

ffibuilder.cdef("""
    struct passwd {
        char *pw_name;
        char *pw_passwd;
        int pw_uid;
        ...;
    };
    struct passwd *getpwuid(int uid);
    struct passwd *getpwnam(const char *name);
""")

if __name__ == "__main__":
    ffibuilder.compile(verbose=True)

Step 2 - Build

$ python demo2_build.py
generating ./_demo2.c
running build_ext
building '_demo2' extension
x86_64-linux-gnu-gcc ... -c _demo2.c -o ./_demo2.o
x86_64-linux-gnu-gcc ... ./_demo2.o -o ./_demo2.cpython-35m-x86_64-linux-gnu.so

$ ls
demo2_build.py  _demo2.c _demo2.o
_demo2.cpython-35m-x86_64-linux-gnu.so

Step 3 - Run

$ cat demo2_run.py
from _demo2 import ffi, lib

# root name
p = lib.getpwuid(0)
print(ffi.string(p.pw_name))

# realitix uid
p = lib.getpwnam(b"realitix")
print(p.pw_uid)

$ python demo2_run.py
b'root'
1000

rebuild

Statistics

Vulkan API
C Header : 5088 loc
Xml description : 6461 loc

C wrapper
Generated C file : 62705 loc
Generator : 1057 py-loc / 1141 c-loc

Cffi wrapper
Generated python file : 4859 loc
Generator : 692 py-loc

How it works ?

1 - Load and parse XML definition

2 - Generate data model

3 - Generate Python module with jinja2

Jinja2 template

C extension

├── converters.c -> 423
├── custom_functions.c -> 103
├── custom_structs.c -> 59
├── extension_functions.c -> 32
├── functions.c -> 8
├── header.c -> 11
├── init.c -> 68
├── init_unions
│   ├── vkclearcolorvalue.c -> 69
│   └── vkclearvalue.c -> 36
├── macros.c -> 111
├── main.c -> 122
├── objects.c -> 99
└── jfilter.py -> 542
Total: 1683

Cffi extension

vulkan.template.py -> 340

Only one small file

Show me the code !

Constants

C extension

PyModule_AddIntConstant(module, {{name}}, {{value}});

Cffi extension

{{name}} = {{value}}

Objects

C extension

new (malloc)

del (free)

init

get (for each member)

Cffi extension

def _new(ctype, **kwargs):
    _type = ffi.typeof(ctype)

    ptrs = {}
    for k, v in kwargs.items():
        # convert tuple pair to dict
        ktype = dict(_type.fields)[k].type
        if ktype.kind == 'pointer':
            ptrs[k] = _cast_ptr(v, ktype)

    init = dict(kwargs,  **{k: v for k, (v, _) in ptrs.items()})
    return ffi.new(_type.cname + '*', init)[0]

Fast API mode

shaderc wrapper

Folder design

├── _cffi_build
│   ├── pyshaderc_build.py
│   └── shaderc.h
├── pyshaderc
│   └── __init__.py
└── setup.py

definition

ffi = FFI()
with open('shaderc.h') as f:
    ffi.cdef(f.read())

building


ffi = FFI()
with open('shaderc.h') as f:
    source = f.read()

ffi.set_source('_pyshaderc', source, libraries=['shaderc_combined'])

if __name__ == '__main__':
    ffi.compile()

use


from pyshaderc._pyshaderc import ffi, lib
        

def compile_into_spirv(raw, stage, suppress_warnings=False):
    # initialize compiler
    compiler = lib.shaderc_compiler_initialize()

    # compile
    result = lib.shaderc_compile_into_spv(compiler, raw, len(raw), stage, b"main")
        

    length = lib.shaderc_result_get_length(result)
    output_pointer = lib.shaderc_result_get_bytes(result)

    tmp = bytearray(length)
    ffi.memmove(tmp, output_pointer, length)
    spirv = bytes(tmp)

    return spirv

setuptools integration


setup(
    ...
    cffi_modules=["_cffi_build/pyshaderc_build.py:ffi"]
)

Demo time !

Give a try to cffi !

@realitix

linagora.com