Python-C api concurrency issue

Question

We are developing a small c server application. The server application does some data processing and responds back to the client. To keep the data processing part configurable and flexible we decided to go for scripting and based on the availability of various ready modules we decided to go for Python. We are using the Python-C api to send/receive the data between c and python.

The Algorithm works something like this:-

1. Server receives some data from client, this data is stored in a dictionary created in c. The dictionary is created using the api function PyDict_New(); from c. The input is stored as a key value pair in the dictionary using the api function PyDict_SetItemString();
2. Next, we execute the python script PyRun_SimpleString(); passing the script as a parameter. This script makes use of the dictionary created in c. Please note, we make the dictionary created in c, accessible to the script using the methods PyImport_AddModule(); and PyModule_AddObject();
3. We store the result of the data processing in the script as a key value pair in the same dictionary created above. The c code can then simply access the result variable(key-value pair) after the script has executed.

The problem The problem we are facing is in the case of concurrent requests coming in from different clients. When multiple requests come in from different clients we tend to object reference count exceptions. Please note, that for each request which comes in for a user, we create an independent dictionary for that user alone. To overcome this problem we encompassed the call to PyRun_SimpleString(); within PyEval_AcquireLock(); and PyEval_ReleaseLock();, but doing this has resulted in the script execution being a blocking call. So if a script is taking long time to execute, all the other users are also waiting for a response.

Could you please suggest the best possible approach or give pointers to where we are going wrong. Please ping me for more information.

Any help/guidance will be appreciated.

Answer 1

也许你错过了这个答案中提到的一个电话。

Answer 2

You should probably read http://docs.python.org/c-api/init.html#thread-state-and-the-global-interpreter-lock Your problem is explained in the first paragraph.

When you acquire the GIL, do so around your direct manipulation of Python objects. The call to PyRun_SimpleString will handle the GIL internally and will give it up on long-running operations or just every X instructions. It WILL NOT be truly multi-threaded, however.

Edit:

You need to acquire the lock and you need to ensure that Python knows it's in a different thread state:

// acquire the lock and switch thread state
PyEval_AcquireLock();
PyThreadState_Swap(perThreadState);

// execute some python code
PyEval_SimpleString("print 123");

// clear the thread state and release the lock
PyThreadState_Swap(NULL);
PyEval_ReleaseLock();

Answer 3

我建议你研究multiprocessing模块。