[英]Line Profiling inner function with Cython
I've had pretty good success using this answer to profile my Cython code, but it doesn't seem to work properly with nested functions. 使用此答案来分析我的Cython代码,我已经取得了很大的成功,但是它似乎无法与嵌套函数一起正常工作。 In this notebook you can see that the profile doesn't appear when the line profiler is used on a nested function.
在此笔记本中,您可以看到在嵌套函数上使用线剖析器时,剖析没有出现。 Is there a way to get this to work?
有没有办法让它工作?
This is seems to be an issue with Cython
, there's a hackish way that does the trick but isn't reliable, you could use it for one-off cases until this issue has been fixed * Cython
似乎是一个问题,有一种Cython
的方法可以解决问题,但并不可靠,您可以将它用于一次性情况,直到解决此问题为止*
line_profiler
source: line_profiler
源: I can't be 100% sure for this but it is working, what you need to do is download the source for line_profiler and go fiddle around in python_trace_callback
. 我不能百分百确定这一点,但是它能正常工作,您需要做的是下载line_profiler的源代码,然后在
python_trace_callback
中python_trace_callback
。 After the code
object is obtained from the current frame of execution ( code = <object>py_frame.f_code
), add the following: 从当前执行框架(
code = <object>py_frame.f_code
)获得code
对象之后,添加以下内容:
if what == PyTrace_LINE or what == PyTrace_RETURN:
code = <object>py_frame.f_code
# Add entry for code object with different address if and only if it doesn't already
# exist **but** the name of the function is in the code_map
if code not in self.code_map and code.co_name in {co.co_name for co in self.code_map}:
for co in self.code_map:
# make condition as strict as necessary
cond = co.co_name == code.co_name and co.co_code == code.co_code
if cond:
del self.code_map[co]
self.code_map[code] = {}
This will replace the code object in self.code_map
with the one currently executing that matches its name and co.co_code
contents. 这会将
self.code_map
的代码对象替换为与其名称和co.co_code
内容匹配的当前正在执行的对象。 co.co_code
is b''
for Cython
, so in essence in matches Cython
functions with that name. co.co_code
是b''
为Cython
,所以在比赛的本质Cython
使用该名称的功能。 Here is where it can become more robust and match more attributes of a code
object (for example, the filename). 在这里它可以变得更加健壮并匹配
code
对象的更多属性(例如,文件名)。
You can then procceed to build it with python setup.py build_ext
and install with sudo python setup.py install
. 然后,您可以使用
python setup.py build_ext
build_ext进行构建,并使用sudo python setup.py install
。 I'm currently building it with python setup.py build_ext --inplace
in order to work with it locally, I'd suggest you do too . 我目前正在使用
python setup.py build_ext --inplace
构建它,以便在本地使用它,我建议您也这样做 。 If you do build it with --inplace
make sure you navigate to the folder containing the source for line_profiler
before import
ing it. 如果确实使用
--inplace
构建它, --inplace
确保在import
之前导航到包含line_profiler
源的文件夹。
So, in the folder containing the built shared library for line_profiler
I set up a cyclosure.pyx
file containing your functions: 因此,在包含为
line_profiler
构建的共享库的文件夹中,我设置了一个包含您的函数的cyclosure.pyx
文件:
def outer_func(int n):
def inner_func(int c):
cdef int i
for i in range(n):
c+=i
return c
return inner_func
And an equivalent setup_cyclosure.py
script in order to build it: 还有一个等效的
setup_cyclosure.py
脚本来构建它:
from distutils.core import setup
from distutils.extension import Extension
from Cython.Build import cythonize
from Cython.Compiler.Options import directive_defaults
directive_defaults['binding'] = True
directive_defaults['linetrace'] = True
extensions = [Extension("cyclosure", ["cyclosure.pyx"], define_macros=[('CYTHON_TRACE', '1')])]
setup(name = 'Testing', ext_modules = cythonize(extensions))
As previously, the build was performed with python setup_cyclosure.py build_ext --inplace
. 和以前一样,构建是通过
python setup_cyclosure.py build_ext --inplace
。
Launching your interpreter from the current folder and issuing the following yields the wanted results: 从当前文件夹启动解释器并发出以下命令,从而得到所需的结果:
>>> import line_profiler
>>> from cyclosure import outer_func
>>> f = outer_func(5)
>>> prof = line_profiler.LineProfiler(f)
>>> prof.runcall(f, 5)
15
>>> prof.print_stats()
Timer unit: 1e-06 s
Total time: 1.2e-05 s
File: cyclosure.pyx
Function: inner_func at line 2
Line # Hits Time Per Hit % Time Line Contents
==============================================================
2 def inner_func(int c):
3 cdef int i
4 1 5 5.0 41.7 for i in range(n):
5 5 6 1.2 50.0 c+=i
6 1 1 1.0 8.3 return c
IPython %%cython
: IPython %%cython
: Trying to run this from IPython
results in an unfortunate situation. 尝试从
IPython
运行此操作会导致不幸的情况。 While executing, the code
object doesn't store the path to the file where it was defined, it simply stored the filename. 执行时,
code
对象不存储定义文件的路径,而只是存储文件名。 Since I simply drop the code
object into the self.code_map
dictionary and since code objects have read-only Attributes, we lose the file path information when using it from IPython
(because it stores the files generated from %%cython
in a temporary directory). 由于我只是将
code
对象放入self.code_map
字典中,并且代码对象具有只读属性,因此从IPython
使用它时,我们会丢失文件路径信息(因为它会将%%cython
生成的文件存储在临时目录中) 。
Because of that, you do get the profiling statistics for your code but you get no contents for the contents. 因此,您的确获得了代码的性能分析统计信息,但没有任何内容。 One might be able to forcefully copy the filenames between the two code objects in question but that's another issue altogether.
一个人也许可以在两个有问题的代码对象之间强行复制文件名,但这是另一个问题。
The issue here is that for some reason, when dealing with nested and/or enclosed functions, there's an abnormality with the address of the code object when it is created and while it is being interpreted in one of Pythons frames. 这里的问题是由于某种原因,在处理嵌套和/或封闭函数时,在创建代码对象以及在Python框架之一中解释代码对象时,其地址存在异常。 The issue you were facing was caused by the following condition not being satisfied :
您遇到的问题是由于不满足以下条件引起的:
if code in self.code_map:
Which was odd. 真奇怪 Creating your function in
IPython
and adding it to the LineProfiler
did indeed add it to the self.code_map
dictionary: 实际上,在
IPython
创建函数并将其添加到LineProfiler
确实确实将其添加到了self.code_map
字典中:
prof = line_profiler.LineProfiler(f)
prof.code_map
Out[16]: {<code object inner_func at 0x7f5c65418f60, file "/home/jim/.cache/ipython/cython/_cython_magic_1b89b9cdda195f485ebb96a104617e9c.pyx", line 2>: {}}
When the time came to actually test the previous condition though, and the current code object was snatched from the current execution frame with code = <object>py_frame.f_code
, the address of the code object was different: 但是,当实际测试先前条件的时间到了,并且当前代码对象是使用
code = <object>py_frame.f_code
从当前执行帧中抢夺的时, code = <object>py_frame.f_code
的地址是不同的:
# this was obtained with a basic print(code) in _line_profiler.pyx
code object inner_func at 0x7f7a54e26150
indicating it was re-created. 表示已重新创建。 This only happens with
Cython
and when a function is defined inside another function. 仅当
Cython
以及在另一个函数中定义一个函数时,才会发生这种情况。 Either this or something that I am completely missing. 这或者我完全不了解的东西。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.