executing c program from a python program

Question

I'm new to programming and was wondering how I can have a python program execute and communicate with ac program. I am doing a mathematical computation in python, and was wondering if I could write up the main computation in C, that way the computation runs faster. I've been reading about "calling c functions from python", "including C or C++ code directly in your Python code", and "using c libraries from python". Is this the same thing? I want a python program to execute ac program and receive the results.

What does it mean to "call C library functions" from python? Would it allow the python script to use c libraries or allow the script to execute code within ac compiler?

thanks

Answer 1

One of the easiest ways to do this is with Cython . You can write code that is very nearly Python, but it compiles down to C. This makes some code (especially numerical computations) much faster. You can also use Cython to easily call C library functions from Python, although ctypes is also adequate.

However, if you actually want to execute a specific C program, you can do so with Subprocess . NB, this is (much) slower than calling a C library or Cython function directly.

Answer 2

There's also numpy which can be reasonably fast when dealing with "array operations" (sometimes called vector operations, but I find that term confusing with SIMD terminology). You'll probably need numpy if you decide to go the cython route, so if the algorithm isn't too complicated, you might want to see if it is good enough with numpy by itself first.

Note that there are two different routes you can take here. You can use subprocess which basically issues system calls to some other program that you have written. This is slow because you need to start a new process and send the data into the process and then read the data back from the process. In other words, the data gets replicated multiple times for each call. The second route is calling a C function from python. Since Cpython (the reference and most common python implementation) is written in C, you can create C extensions. They're basically compiled libraries that adhere to a certain API. Then Cpython can load those libraries and use the functions inside, passing pointers to the data. In this way, the data isn't actually replicated -- You're working with the same block of memory in python that you're using in C. The downside here is that the C API is a little complex. That's where 3rd party extensions and existing libraries come in (numpy, cython, ctypes, etc). They all have different ways of pushing computations int C functions without you having to worry about the C API. Numpy removes loops so you can add, subtract, multiply arrays quickly (among MANY other things). Cython translates python code to C which you can then compile and import -- typically to gain speed here you need to provide additional hints which allow cython to optimize the generated code, ctypes is a little fragile since you have to re-specify your C function prototype, but otherwise it's pretty easy as long as you can compile your library into a shared object ... The list could go on.

Also note that if you're not using numpy, you might want to check out pypy. It claims to run your python code faster than Cpython.

Answer 3

Options that come to mind:

1. Cython: A dialect of Python that can freely mix Python and C datatypes. Gives a modest speedup with a few type annotations for classes, and can be quite a bit faster if you give your main loop all C types to operate on. Pretty much CPython only at this time, but there's been discussion of making it work with Pypy.
2. SWIG: A sort of interface/glue language for matching up a C library with a potentially large list of higher level languages, including CPython.
3. ctypes: Allows you to call individual C functions and access individual C datatypes. Not blazing in CPython unless you spend a lot of time in the C code. Can be a bit brittle. Also works on Pypy.
4. C extension module: Pretty standard stuff for CPython - this is how much of the CPython the standard library is put together. Some but not all of these work with Pypy if you use cpyext. In short, with this you're writing a module with a bunch of boiler plate, in C, but it can be called from CPython.
5. CFFI: A new foreign function interface coming from the Pypy project. They have it working with CPython now, and intend to get it going with Pypy before long. It is like ctypes, but is less brittle - the downside is it requires a C compiler at runtime, and it's pretty young code.
6. Pypy: This isn't really a way calling C itself, but it's got almost as much speed as C for a lot of pure python code. If you don't have a lot of C extension modules you require, then Pypy might be a good option for you.
7. subprocess: A very portable way of interacting with another process in not-necessarily-the-same-language. Can be used with just about any language in the child process. Not blazing fast, unless you spend little time passing data back and forth. Pretty much all data exchanged has to be serialized to ASCII or something, but it is so portable and simple and loosely coupled that it's worth thinking about.
8. You might be able to just optimize your Python: http://wiki.python.org/moin/PythonSpeed/PerformanceTips

Don't rush to coding in C - it's much more expensive in terms of programmer time and is prone to subtle, hard to find bugs. Run your code in Pure Python first, to get the program produces correct results at whatever speed, and then if you find that the program is too slow, profile the program to decide which part(s) need to be redone in one of the ways listed above. Usually you'll only need 0-2% of your program to be done in something other than pure python - may as well reap the programmer-time savings for as much of the code as is practical.

Answer 4

You want ctypes . It lets you run C functions from Python directly without any crazy extension stuff. Of course, you should make sure you're not optimizing prematurely. Python's math functions are pretty fast.

The ctypes module lets you call C library functions directly from Python code. That means it lets you drop the compiled library somewhere that Python can get to it and run those compiled functions. This is not the same as calling C programs from Python which, as Conrad has pointed out, can be done with subprocess. Of course, subprocess doesn't need programs written in C. It can run any executable.