In this Python Tutorial, we will be discussing the ctypes library. Often when programming in Python, we may feel the need to use functions that may be written in another language. This is most commonly because Python is much slower in comparison to languages like C++ or C.
There are actually many libraries in Python that use functions written in C or C++, such as NumPy or OpenCV. Importing libraries like this, and using their functions is made possible with the ctypes library in Python.
Let’s explore how you can do this as well in this Python ctypes Tutorial!
Importing a C library with ctypes
Now let’s try to import a basic library with a single function over to our Python program using ctypes. You can’t link your regular .c file though. You need to generate a shared library, which can be done with the following command.
gcc -fPIC -shared -o clibrary.so clibrary.c“clibrary” is the name we gave to our C file. The name itself can be anything, just be mindful of the extensions when using this command.
We can also generate a shared library using a .cpp file instead, but there are a few differences in how it is generated, and how we write our C/C++ code. We will discuss this in further depth at the end of this tutorial with an example or two.
Let’s first create our .c file.
#include <stdio.h>
void prompt()
{
printf("hello world\n");
}
Now we need to go over to our Python file, import the ctypes library and then use the CDILL function to load up the shared library file. Remember to include the full path to the shared library file if it’s not in the same directory as the Python file.
import ctypes
libObject = ctypes.CDLL('clibrary.so')
The CDILL function returns a library object, which we can use to access functions within the library.
Calling C Functions from Python
Now that we have a library object, we can call the function from that library as a method of the library object.
libObject = ctypes.CDLL('clibrary.so')
libObject.prompt()
hello worldAs you can see, the above function has executed successfully. Now let’s take at some more advanced examples, where we call C functions from Python.
Let’s modify a prompt a bit, so that it prints out some variables that we pass into it.
#include <stdio.h>
void prompt(int num)
{
printf("The number %d was entered", num);
}
Now let’s try calling this function from Python by passing an integer into it. Remember to generate a new .so file after making any changes to the .c library.
import ctypes
testlib = ctypes.CDLL('clibrary.so')
testlib.prompt(20)
The number 20 was enteredIt works!
Using Function Signatures to Call Functions
There are alternate techniques that we can use to call functions that we will discuss in this Python ctypes tutorial.
Let’s create a function in our C Library, that adds two numbers together, and returns the result. A very basic addition function.
int add(int num1, int num2) {
return num1 + num2;
}
Now for our Python file. What you need to do first, is acquire the function signature of the C function. Like so.
addTwoNumbers = clibrary.add
You can even keep the same name if you want. So add = clibrary.add is also valid (thanks to namespaces). Now we need to define the function parameters, and function return type.
addTwoNumbers.argtypes = [ctypes.c_int, ctypes.c_int]
addTwoNumbers.restype = ctypes.c_int
argtypes is for the parameters, and restype is for the return type. Notice how the argtypes takes a list, as there can be multiple parameters. On the other hand, restype takes a single value, as there can only be one return type or none.
The complete code:
import ctypes
clibrary = ctypes.CDLL('clibrary.so')
addTwoNumbers = clibrary.add
addTwoNumbers.argtypes = [ctypes.c_int, ctypes.c_int]
addTwoNumbers.restype = ctypes.c_int
print("Sum of two numbers is :", addTwoNumbers(20, 10))
Sum of two numbers is : 30Strings in ctypes
A common issue that people run into, is when dealing with strings in C and Python. The problem occurs, because in Python, strings are immutable which means they cannot be changed. They can only be overwritten completely, not modified. in C and C++, strings
First let’s try passing a string to a function in C.
#include <stdio.h>
void display(char* str) {
printf("%s", str);
}
Here is out simple C function, which will print out the string we pass to it.
import ctypes
clibrary = ctypes.CDLL('clibrary.so')
clibrary.display(b"John")
JohnOur python code here, simply passes a string to the C function. Notice that little “b” before the string? That is done to declare it as binary, which is nessacery in order for it to be compatible with C.
Another rather important thing to mention, is how to convert a variable containing a regular string (not a binary string), to a C acceptable string. We can’t use that little “b” unless we are dealing with raw strings. But for variables, we have encode method which we can use.
import ctypes
clibrary = ctypes.CDLL('func.so')
cstring = "John"
clibrary.display(cstring.encode())
Attempting to do this without the encode() method will return an error!
Supported and Unsupported Datatypes
Now that we have this working, let’s move try and modify the string, within the C function.
void increment(char* str) {
for (int i = 0; str[i] != 0; i++) {
str[i] = str[i] + 1;
}
}
This C function will increment each character in the string by 1. So if there is an “A” character, it will become a “B” due to the ascii codes. Let’s try passing a string from Python, into this function.
import ctypes
clibrary = ctypes.CDLL('clibrary.so')
string = "Hello"
print("Before:" , string)
clibrary.increment(string)
print("After: ", cstring)
Before: Hello
After: HelloAs you can see from the output, there was no change. This is because Python strings are not compatible with C. Now let’s resolve this issue using some of ctypes special datatypes which are compatible with C.
Do try and use ctypes datatypes as much as possible instead of Python types when writing code which is meant to interact with C. Very few Python types can directly be used with C (without any errors), among-st which two are integers and bytes (byte strings).
import ctypes
clibrary = ctypes.CDLL('func.so')
cstring = ctypes.c_char_p(b"Hello")
print("Before:" , cstring.value)
clibrary.increment(cstring)
print("After: ", cstring.value)
Before: b'Hello'
After: b'Ifmmp'Instead of using a Python string, we use a char pointer (the string equivalent) from ctypes. Hence it worked! Every ctype datatype has a value attribute, which returns a native python object. Hence we were able to return a string from the char pointer (which is basically C’s version of a string).
Alternatively, we just use cstring = b”Hello”, as Python bytes are also supported by C..
Creating Mutable memory with ctypes String Buffers
When using character pointers, we aren’t exactly dealing with mutable memory. The memory in question was still immutable. If we try assigning a new value to it, a new memory location will be assigned to the string with that value. Instead of the old memory location being assigned a new value.
This can cause problems with functions which expect to receive mutable memory. The below example illustrates this issue.
import ctypes
clibrary = ctypes.CDLL('clibrary.so')
cstring = ctypes.c_char_p(b"Hello")
print("Before:" , cstring, cstring.value)
cstring.value = b"World"
print("After: ", cstring, cstring.value)
Before: c_char_p(2186232967920) b'Hello'
After: c_char_p(2186232970992) b'World'Notice how the address is different after the assignment? The address is the value in the brackets.
To resolve this we create String buffers using ctypes. Let’s take a look at how to do this.
clibrary = ctypes.CDLL('clibrary.so')
cstring = ctypes.create_string_buffer(b"Hello")
print("Before:" , cstring, cstring.value)
clibrary.increment(cstring)
print("After: ", cstring, cstring.value)
Before: c_char_p(2200599049040) b'Hello'
After: c_char_p(2200599049040) b'Ifmmp'As you can see, the string was properly modified. Creating a string buffer with ctypes, gives us mutable memory which we can use with C functions.
You can also choose to pass in an integer “n” instead of a string into the create_string_buffer() function which will create an empty string buffer of “n” size.
Bonus Advice
If you have turned to ctypes in the hope of speeding up your Python code, by writing some of it in C/C++, there is another option to consider. There is a popular Python library called “Cython” which can add certain C/C++ features to Python such as “static type checking” which can dramatically speed up your program by around 5x – 1000x times! (depending on the task).
You can learn more about it by reading this article on Cython, complete with performance benchmarks as well.
Memory Management with Pointers in ctypes
Memory management with Pointers in ctypes is the last topic of this tutorial.
C++ and C both make use of Pointers, but Pointers are not available as a datatype in Python. The ctypes library gives ctypes.POINTER, which we can use to create a pointer in Python.
Let’s take a look at some examples using this ctypes pointer.
#include <stdio.h>
#include <string.h>
#include <stdlib.h>
char* alloc_memory(void) {
char* str = strdup("Hello World");
printf("Memory allocated...\n");
return str;
}
void free_memory(char* ptr) {
printf("Freeing memory...\n");
free(ptr);
}
First we declared two functions in our C library, one for allocating memory and one for freeing memory. The alloc_memory() function returns a pointer to the string that we created. And the free_memory() function frees it up.
The basic gist of it, that we will use the alloc_memory() function to give us the string. We will then use this string in Python for whatever purpose we wish, then use the free_memory() function to free up the memory.
import ctypes
clibrary = ctypes.CDLL('clibrary.so')
# Defining alloc function and it's return type
alloc_func = clibrary.alloc_memory
alloc_func.restype = ctypes.POINTER(ctypes.c_char)
# Defining free function and it's return type
free_func = clibrary.free_memory
free_func.argtypes = [ctypes.POINTER(ctypes.c_char)]
# Using the function to return a string
cstring_pointer = alloc_func()
str = ctypes.c_char_p.from_buffer(cstring_pointer)
print(str.value)
# Freeing memory
free_func(cstring_pointer)
Memory allocated...
b'Hello World'
Freeing memory...Creating Pointers with ctypes
There is an alternative way to create a pointer which you might find useful. It’s generally a bit slower than using ctypes.POINTER however. (Due to an inbuilt system that allows ctypes.POINTER to reuse old Pointers)
ctypes.POINTER() takes as parameter the type of pointer you wish to create, an integer pointer for example. ctypes.pointer() takes as parameter an object, which it then returns a pointer to. The pointer() however, only accepts datatypes from the ctypes module. So regular python datatypes will not work.
num1 = ctypes.c_int(100)
num2 = ctypes.c_int(200)
ptr = ctypes.pointer(num1)
print(ptr.contents.value)
ptr.contents = num2
print(ptr.contents.value)
100
200You can use the contents attribute to access the object to which the Pointer points. You can further access the value attribute to print out it’s actual python value, rather the a ctype object.
Pointers also play a very important role when passing arrays back and forth. As you might already know, you cannot directly pass an entire array in C as an argument or return type. You need to use pointers instead.
Did you know you can also pass Custom Classes/Structs back and forth between your C and Python program using ctypes? Follow this link for more information if you are interested.
Using C++ with ctypes
This may not be very obvious, but you can also use C++ with Python with the help of ctypes. It is a bit trickier and there are some limitations but it’s pretty simple to grasp.
The shared library is generated in the same manner, but with .cpp instead .c as the extension and g++ instead of gcc.
g++ -fPIC -shared -o cppLibrary.so cppLibrary.cppNow, there are a few things to keep in mind. C++ supports Function overloading, as a result of which C++ also performs name mangling on it’s function names. What this means, is that additional information will be appended to the function name, which results in function name being changed. This is done to have a unique name for each function for the compiler to use.
Name mangling is performed on every function, regardless of whether it is an overloaded function or not. So you can’t try avoid it. If you try to call a C++ function from a C++ Shared library in Python using ctypes, it will give an error that the function was not found.
#include <iostream>
void display() {
std::cout << "Hello world\n";
}
import ctypes
clibrary = ctypes.CDLL('cppLibrary.so')
clibrary.display()
AttributeError: function 'display' not foundAs you can see, it’s not working. Luckily, it has an easy fix! All you need to do is wrap it inside extern “C” as shown below. This disables name mangling, and thus allows the code to run using ctypes.
#include <iostream>
extern "C" {
void display() {
std::cout << "Hello world\n";
}
}
import ctypes
clibrary = ctypes.CDLL('cppLibrary.so')
clibrary.display()
Hello worldNow it works!
Name mangling and it’s effects on using C++ with Python are a separate topic completely, and worth researching on. This section describes the main process of using C++ in Python, but there are several good practices that you also need to be learning.
And that’s the end of our Python ctypes tutorial!
This marks the end of the Python ctypes Tutorial. Any suggestions or contributions for CodersLegacy are more than welcome. Questions regarding the article content can be asked in the comments section below.
