Getting errors when trying to run Pyflame inside Python script

Question

The aim: profiling a Python script with Pyflame in another Python script.

The details: code is run on a virtual machine with Ubuntu 14.04 LTS.

1) To run the profiled script I'm using:

process = subprocess.Popen(["python", "python_script.py"])

2) Then I'm trying to attach Pyflame to the process (treat this line as a smoke test):

subprocess.Popen(["sudo pyflame -s 60 -r 0.0001 " + str(process.pid)],
                      shell=True)

The problem: Without sudo in the above statement I'm getting a Failed to attach to PID ...: Operation not permitted error.

With sudo I'm getting Failed to locate libpython named libpython2.7.so .

I would be grateful for any ideas regarding how to make statement 2) work, with or without sudo !

Answer 1

Apparently, Pyflame will return the second error that you got if these conditions are met:

• You compiled Pyflame for Python 2.X.
• You try to profile a script running under a Python 3.X interpreter.

Take a look at the source code of Pyflame here and here . When you run Pyflame, it looks into the memory space of your interpreter and it tries to find libpython2.7.so . For a Python 3.X interpreter, the library is not there.

So I suggest that you try the following:

1. Run the following command. You'll confirm whether the library is properly loaded in your interpreter. cat /proc/{YOUR_PID}/maps | grep libpython2.7
2. If it's not, then consider the following:
   1. Maybe you profiled the wrong process (ie it's not a Python process).
   2. Maybe it's a Python 3.X process, in which case you'll need to re-compile Pyflame to target Python 3.X.

To target Python 3.X when compiling Pyflame, refer to the documentation. Hint: you'll need to change the call to ./configure as follows:

./configure --with-python=python3

Besides, if you have trouble making python3 recognized as a library in your environment, you'll need to symlink to the Python 3.X interpreter's executable. This is a bit out of scope, so you can check out my personal blog for more details (see the post "Using Uber's Pyflame and Logs to Tackle Scaling Issues"). Disclaimer: I'm the author of that post.