Introduction
Inspired by the
article
"Embedding
Python
in
Multi-Threaded C/C++
Applications" (Linux
Journal), I felt the
need for a more
comprehensive coverage on
the topic of embedding
Python. While
writing this
article, I had
two objectives:
-
This is written for
programmers who are more
experienced in C/C++
than in
Python,
the
tutorial
takes a practical
approach and omits all
theoretical discussions.
-
Try to maintain
Python's
cross-platform
compatibility when
writing the embedding
code.
Now, you have got some
modules written in
Python by others
and you want to use them.
You are experienced in C/C++,
but fairly new to
Python. You might
wonder if you can find a
tool to convert them to C
code, like the conversion
from FORTRAN. The answer is
no. Some tools can help you
generate the executable from
a Python module. Is the
problem solved? No.
Converting code into
executables usually makes
things even more
complicated, as you must
figure out how your C/C++
application
communicates with the
executable "black-box".
I am going to introduce C/C++
programmers to
Python/C API, a C
library that helps to embed
python modules
into C/C++
applications. The
API library provides a bunch
of C routines to initialize
the
Python
Interpreter, call
into your
Python modules
and finish up the embedding.
The library is built with
Python and distributed with
all the recent
Python releases.
Part I of this article
series discusses the basics
of
Python embedding.
Part II will move on to more
advanced topics. This
tutorial does not teach the
Python language
systematically, but I will
briefly describe how the
Python code works
when it comes up. The
emphasis will be on how to
integrate
Python modules
with your C/C++
applications. See
article:
"Embedding
Python in C/C++:
Part II".
In order to use the
source code, you
should install a recent
Python release,
Visual C++ (or
GCC compiler on Linux). The
environment that I have used
to test is:
Python 2.4
(Windows and Linux),
Visual C++ 6.0
(Windows) or GCC 3.2 (RedHat
8.0 Linux). With
Visual C++,
select the Release
configuration to build.
Debug configuration requires
Python debug library "python24_d.lib",
which is not delivered with
normal distributions.
Background
Python is a
powerful interpreted
language, like Java, Perl
and PHP. It supports a long
list of great features that
any programmer would expect,
two of my favorite features
are "simple" and "portable".
Along with the available
tools and libraries,
Python makes a
good language for modeling
and simulation developers.
Best of all, it's free and
the tools and libraries
written for
Python
programmers are also free.
For more details on the
language, visit the official
website.
Embedding basics: functions,
classes and methods
First, let us start from a
sample C program that calls
a function within a
Python module.
Here is the source file "call_function.c":
Collapse
Copy Code
#include <Python.h>
int main(int argc, char *argv[])
{
PyObject *pName, *pModule, *pDict, *pFunc, *pValue;
if (argc < 3)
{
printf("Usage: exe_name python_source function_name\n");
return 1;
}
Py_Initialize();
pName = PyString_FromString(argv[1]);
pModule = PyImport_Import(pName);
pDict = PyModule_GetDict(pModule);
pFunc = PyDict_GetItemString(pDict, argv[2]);
if (PyCallable_Check(pFunc))
{
PyObject_CallObject(pFunc, NULL);
} else
{
PyErr_Print();
}
Py_DECREF(pModule);
Py_DECREF(pName);
Py_Finalize();
return 0;
}
The
Python source
file "py_function.py"
is as follows:
Collapse
Copy Code
'''py_function.py - Python source designed to '''
'''demonstrate the use of python embedding'''
def multiply():
c = 12345*6789
print 'The result of 12345 x 6789 :', c
return c
Note that checks for the
validity of objects are
omitted for brevity. On
Windows, simply compile the
C source and get the
executable, which we call "call_function.exe".
To run it, enter the command
line "call_function
py_function multiply
".
The second argument is the
name of the Python file
(without extension), which
when loaded becomes the
module name. The third
argument is the name of the
Python function you are
going to call within the
module. The Python source
does not take part in
compiling or linking; it's
only loaded and interpreted
at run-time. The output from
the execution is:
Collapse
Copy Code
The result of 12345 x 6789 : 83810205
The C code itself is
self-explanatory, except
that:
-
Everything in Python is
an object.
pDict
and
pFunc
are borrowed references
so we don't need to
Py_DECREF()
them.
-
All the Py_XXX and
PyXXX_XXX calls are
Python/C
API calls.
-
The code will compile
and run on all the
platforms that
Python
supports.
Now, we want to pass
arguments to the
Python function.
We add a block to handle
arguments to the call:
Collapse
Copy Code
if (PyCallable_Check(pFunc))
{
if( argc > 3 )
{
pArgs = PyTuple_New(argc - 3);
for (i = 0; i < argc - 3; i++)
{
pValue = PyInt_FromLong(atoi(argv[i + 3]));
if (!pValue)
{
PyErr_Print();
return 1;
}
PyTuple_SetItem(pArgs, i, pValue);
}
pValue = PyObject_CallObject(pFunc, pArgs);
if (pArgs != NULL)
{
Py_DECREF(pArgs);
}
} else
{
pValue = PyObject_CallObject(pFunc, NULL);
}
if (pValue != NULL)
{
printf("Return of call : %d\n", PyInt_AsLong(pValue));
Py_DECREF(pValue);
}
else
{
PyErr_Print();
}
}
The new C source adds a
block of "Prepare the
argument list for the call"
and a check of the returned
value. It creates a tuple
(list-like type) to store
all the parameters for the
call. You can run the
command "call_function
py_source multiply1
6
7
"
and get the output:
Collapse
Copy Code
The result of 6 x 7 : 42
Return of call : 42
Writing classes in
Python is easy.
It is also easy to use a
Python class in
your C code. All you need to
do is create an instance of
the object and call its
methods, just as you call
normal functions. Here is an
example:
Collapse
Copy Code
#include <Python.h>
int main(int argc, char *argv[])
{
PyObject *pName, *pModule, *pDict,
*pClass, *pInstance, *pValue;
int i, arg[2];
if (argc < 4)
{
printf(
"Usage: exe_name python_fileclass_name function_name\n");
return 1;
}
pClass = PyDict_GetItemString(pDict, argv[2]);
if (PyCallable_Check(pClass))
{
pInstance = PyObject_CallObject(pClass, NULL);
}
if( argc > 4 )
{
for (i = 0; i < argc - 4; i++)
{
arg[i] = atoi(argv[i + 4]);
}
pValue = PyObject_CallMethod(pInstance,
argv[3], "(ii)", arg[0], arg[1]);
} else
{
pValue = PyObject_CallMethod(pInstance, argv[3], NULL);
}
if (pValue != NULL)
{
printf("Return of call : %d\n", PyInt_AsLong(pValue));
Py_DECREF(pValue);
}
else
{
PyErr_Print();
}
}
The third parameter to
PyObject_CallMethod()
,
"(ii)
"
is a format string, which
indicates that the next
arguments are two integers.
Note that
PyObject_CallMethod()
takes C variables types as
its arguments, not Python
objects. This is different
from the other calls we have
seen so far. The Python
source "py_class.py"
is as follows:
Collapse
Copy Code
'''py_class.py - Python source designed to demonstrate'''
'''the use of python embedding'''
class Multiply:
def __init__(self):
self.a = 6
self.b = 5
def multiply(self):
c = self.a*self.b
print 'The result of', self.a, 'x', self.b, ':', c
return c
def multiply2(self, a, b):
c = a*b
print 'The result of', a, 'x', b, ':', c
return c
To run the application, you
add a class name between the
module and function names,
which is "Multiply
"
in this case. The command
line becomes "call_class
py_class Multiply multiply
"
or "call_class
py_class Multiply multiply2
9
9
".
Multi-threaded Python
embedding
With the above preparations,
we are ready for some
serious business. The
Python module and
your C/C++
application have
to run simultaneously from
time to time. This is not
uncommon in simulation
communities. For instance,
the
Python module
to be embedded is part of a
real-time simulation and you
run it in parallel with the
rest of your simulation.
Meanwhile, it interacts with
the rest at run-time. One
conventional technique is
multi-threading. There are a
number of options for
multi-threaded embedding. We
are going to discuss two of
them here.
In one approach you create a
separate thread in C and
call the
Python module
from the thread function.
This is natural and correct,
except that you need to
protect the Python
Interpreter state.
Basically, we lock the
Python
Interpreter before you use
it and release after the use
so that
Python can track
its states for the different
calling threads.
Python provides
the global lock for this
purpose. Let us look at some
source code
first. Here is the complete
content of "call_thread.c":
Collapse
Copy Code
#include <Python.h>
#include <stdlib.h>
#include <stdio.h>
#include <string.h>
#include <errno.h>
#ifdef WIN32 #include <Windows.h>
#include <process.h>
#define sleep(x) Sleep(1000*x)
HANDLE handle;
#else #include <pthread.h>
pthread_t mythread;
#endif
void ThreadProc(void*);
#define NUM_ARGUMENTS 5
typedef struct
{
int argc;
char *argv[NUM_ARGUMENTS];
} CMD_LINE_STRUCT;
int main(int argc, char *argv[])
{
int i;
CMD_LINE_STRUCT cmd;
pthread_t mythread;
cmd.argc = argc;
for( i = 0; i < NUM_ARGUMENTS; i++ )
{
cmd.argv[i] = argv[i];
}
if (argc < 3)
{
fprintf(stderr,
"Usage: call python_filename function_name [args]\n");
return 1;
}
#ifdef WIN32
handle = (HANDLE) _beginthread( ThreadProc,0,&cmd);
#else
pthread_create( &mythread, NULL,
ThreadProc, (void*)&cmd );
#endif
for(i = 0; i < 10; i++)
{
printf("Printed from the main thread.\n");
sleep(1);
}
printf("Main Thread waiting for My Thread to complete...\n");
#ifdef WIN32
WaitForSingleObject(handle,INFINITE);
#else
pthread_join(mythread, NULL);
#endif
printf("Main thread finished gracefully.\n");
return 0;
}
void ThreadProc( void *data )
{
int i;
PyObject *pName, *pModule, *pDict,
*pFunc, *pInstance, *pArgs, *pValue;
PyThreadState *mainThreadState, *myThreadState, *tempState;
PyInterpreterState *mainInterpreterState;
CMD_LINE_STRUCT* arg = (CMD_LINE_STRUCT*)data;
for(i = 0; i < 15; i++)
{
printf("...Printed from my thread.\n");
sleep(1);
}
Py_Initialize();
PyEval_InitThreads();
mainThreadState = PyThreadState_Get();
mainInterpreterState = mainThreadState->interp;
myThreadState = PyThreadState_New(mainInterpreterState);
PyEval_ReleaseLock();
PyEval_AcquireLock();
tempState = PyThreadState_Swap(myThreadState);
pName = PyString_FromString(arg->argv[1]);
pModule = PyImport_Import(pName);
pDict = PyModule_GetDict(pModule);
pFunc = PyDict_GetItemString(pDict, arg->argv[2]);
if (PyCallable_Check(pFunc))
{
pValue = PyObject_CallObject(pFunc, NULL);
}
else {
PyErr_Print();
}
Py_DECREF(pModule);
Py_DECREF(pName);
PyThreadState_Swap(tempState);
PyEval_ReleaseLock();
PyThreadState_Clear(myThreadState);
PyThreadState_Delete(myThreadState);
Py_Finalize();
printf("My thread is finishing...\n");
#ifdef WIN32
_endthread();
#else
pthread_exit(NULL);
#endif
}
The thread function needs a
bit of explanation.
PyEval_InitThreads()
initializes Python's thread
support.
PyThreadState_Swap(myThreadState)
swaps in the state for the
current thread, and
PyThreadState_Swap(tempState)
swaps it out. The Python
Interpreter will save what
happens between the two
calls as the state data
related to this thread. As a
matter of fact, Python saves
the data for each thread
that is using the
Interpreter so that the
thread states are mutually
exclusive. But it is your
responsibility to create and
maintain a state for each C
thread. You may wonder why
we didn't call the first
PyEvel_AcquireLock()
.
Because
PyEval_InitThreads()
does so by default. In other
cases, we do need to use
PyEvel_AcquireLock()
and
PyEvel_ReleaseLock()
in pairs.
Run "call_thread
py_thread pythonFunc
"
and you can get the output
as shown below. The file "py_thread.py"
defines a function called
pythonFunc()
in which a similar random
testing block prints "print
from pythonFunc..." to
screen fifteen times.
Collapse
Copy Code
Printed from the main thread.
...Printed from my thread.
Printed from the main thread.
...Printed from my thread.
Printed from the main thread.
...Printed from my thread.
Printed from the main thread.
...Printed from my thread.
Printed from the main thread.
...Printed from my thread.
...Printed from my thread.
Printed from the main thread.
Printed from the main thread.
...Printed from my thread.
Printed from the main thread.
...Printed from my thread.
Printed from the main thread.
...Printed from my thread.
Printed from the main thread.
...Printed from my thread.
Main Thread waiting for My Thread to complete...
...Printed from my thread.
...Printed from my thread.
...Printed from my thread.
...Printed from my thread.
...Printed from my thread.
My thread is finishing...
Main thread finished gracefully.
Obviously, the
implementation is getting
complicated because writing
multi-threading code in C/C++
is not trivial. Although the
code is portable, it
contains numerous patches,
which require detailed
knowledge of the system call
interfaces for specific
platforms. Fortunately,
Python has done most of this
for us, which brings up the
second solution to our
question under discussion,
namely, letting Python
handle the multi-threading.
This time the
Python code is
enhanced to add a threading
model:
Collapse
Copy Code
''' Demonstrate the use of python threading'''
import time
import threading
class MyThread(threading.Thread):
def __init__(self):
threading.Thread.__init__(self)
def run(self):
for i in range(15):
print 'printed from MyThread...'
time.sleep(1)
def createThread():
print 'Create and run MyThread'
background = MyThread()
background.start()
print 'Main thread continues to run in foreground.'
for i in range(10):
print 'printed from Main thread.'
time.sleep(1)
print 'Main thread joins MyThread and waits until it is done...'
background.join() # Wait for the background tucancode.net to finish
print 'The program completed gracefully.'
The C code does not handle
threading any more. All it
needs to do is just call
createThread()
.
Try to use the previous "call_function.c".
You can run "call_function
py_thread createThread
"
to see how the output looks
like. In this case, the
second solution is cleaner
and simpler. More
importantly,
Python threading
model is portable. While the
C threading code for Unix
and Windows is different, it
remains the same in
Python.
The Python function
createThread()
is not required if our C
code calls the thread class'
start()
and
joint()
methods. The relevant
changes are listed in the
following (from the C source
file "call_thread_2.c"):
Collapse
Copy Code
pInstance = PyObject_CallObject(pClass, NULL);
PyObject_CallMethod(pInstance, "start", NULL);
i = 0;
while(i<10)
{
printf("Printed from C thread...\n");
PyObject_CallMethod(pInstance, "join", "(f)", 0.001);
Sleep(1000);
i++;
}
printf(
"C thread join and wait for Python thread to complete...\n");
PyObject_CallMethod(pInstance, "join", NULL);
printf("Program completed gracefully.\n");
Basically, after you create
the class instance, call its
start()
method to create a new
thread and execute its
run()
method. Note that without
frequent short joins to the
created thread, the created
thread can only get executed
at the beginning and the
main thread will not release
any CPU to it until it's
done. You may try this out
by commenting the joint call
within the
while
loop. The behavior is
somehow different from that
of the previous case where
we had called
start()
from within the Python
module. This seems to be a
feature of multi-threading
which is not documented in
the
Python Library Reference.
Points of interest
I have deliberately paid
attention to writing
generic, portable C code for
Python embedding.
By encapsulating low-level
system calls,
Python supports
platform portability and
makes writing portable code
easier. Most
Python modules
can be ported between
Unix-like environment and
Windows with little effort.
We should keep this
portability in mind when
writing C/C++ wrapping code
for
Python
modules. It may not be
always straightforward to
write portable C code
yourself.
Python has done a
lot of hard work, as in the
above case. Try to explore
easier, simpler and cleaner
solutions. Anyway, I stop
here. How to write portable
Python code goes beyond the
scope of this tutorial. It
could well make a good title
for a new article.
While embedding is a good
option for utilizing
Python modules
in
C/C++
applications,
there are other alternative
approaches. On Windows, some
tools (e.g. "py2exe") can
convert Python modules into
Windows executables
directly. Then, you can
spawn a process to run the
executable from within the
C/C++
application. One
drawback is that you cannot
call the module directly.
Instead, your application
has to interact with the
module through certain types
of IPC. This requires that
the Python module under
discussion be "IPC ready",
meaning that it should have
implemented an IPC interface
to process incoming and
outgoing data. Part II of
this article will discuss
embedding-related IPC
techniques.
All the source code provided
in this
article is simple
C code for demonstration
purpose. In practice, I
recommend to put the
Python embedding
code in C++ wrapper classes.
This way, high-level
application developers don't
have to deal with the
embedding details.
Conclusion
In this part, I have covered
Python embedding from the
basics such as calling
functions, classes and
methods, to not so basic
topics like multi-threaded
embedding. Python/C API
provides a consistent
calling interface to ease
the tucancode.net of integration
between C/C++
and
Python
modules.
The discussion of
multi-threaded embedding has
raised a question for us:
how does your C/C++
application
communicate with the
embedded Python module? The
second part of this
article shall
address this issue from the
IPC point of view.