[英]Python memory_profiler: @profile not working on multithreading
I have the following code from the example
folder with the exception that I added @profile
.我从example
文件夹中获得了以下代码,但添加了@profile
。 I am just trying to make this example run because in my code which is more complex I have the same error and I would like to know how much memory is used on each line.我只是想让这个例子运行,因为在我更复杂的代码中我有同样的错误,我想知道每行使用了多少 memory。
SYSTEM:系统:
Python: 3.9 Python:3.9
memory-profiler: 0.58内存分析器:0.58
OS: Manjaro操作系统:Manjaro
CODE:代码:
import time
import multiprocessing as mp
from memory_profiler import profile
# Big numbers
X6 = 10 ** 6
X7 = 10 ** 7
def worker(num, wait, amt=X6):
"""
A function that allocates memory over time.
"""
frame = []
for idx in range(num):
frame.extend([1] * amt)
time.sleep(wait)
del frame
def main_sequential():
"""
A sequential version of the work, where one worker is called at a time.
"""
worker(5, 5, X6)
worker(5, 2, X7)
worker(5, 5, X6)
worker(5, 2, X7)
@profile
def main_multiproc():
"""
A multiprocessing version of the work, where workers work in their own
child processes and are collected by the master process.
"""
pool = mp.Pool(processes=4)
tasks = [
pool.apply_async(worker, args) for args in
[(5, 5, X6), (5, 2, X7), (5, 5, X6), (5, 2, X7)]
]
results = [p.get() for p in tasks]
if __name__ == '__main__':
main_multiproc()
RUN COMMAND:运行命令:
python -m memory_profiler MyScript.py
ERROR:错误:
Traceback (most recent call last):
File "/usr/lib/python3.9/runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/lib/python3.9/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/home/vlad/Documents/TradingBotRL/venv/lib/python3.9/site-packages/memory_profiler.py", line 1303, in <module>
exec_with_profiler(script_filename, prof, args.backend, script_args)
File "/home/vlad/Documents/TradingBotRL/venv/lib/python3.9/site-packages/memory_profiler.py", line 1204, in exec_with_profiler
exec(compile(f.read(), filename, 'exec'), ns, ns)
File "MyScript.py", line 47, in <module>
main_multiproc()
File "/home/vlad/Documents/TradingBotRL/venv/lib/python3.9/site-packages/memory_profiler.py", line 1142, in wrapper
val = prof(func)(*args, **kwargs)
File "/home/vlad/Documents/TradingBotRL/venv/lib/python3.9/site-packages/memory_profiler.py", line 717, in f
return func(*args, **kwds)
File "MyScript.py", line 43, in main_multiproc
results = [p.get() for p in tasks]
File "MyScript.py", line 43, in <listcomp>
results = [p.get() for p in tasks]
File "/usr/lib/python3.9/multiprocessing/pool.py", line 771, in get
raise self._value
File "/usr/lib/python3.9/multiprocessing/pool.py", line 537, in _handle_tasks
put(task)
File "/usr/lib/python3.9/multiprocessing/connection.py", line 211, in send
self._send_bytes(_ForkingPickler.dumps(obj))
File "/usr/lib/python3.9/multiprocessing/reduction.py", line 51, in dumps
cls(buf, protocol).dump(obj)
_pickle.PicklingError: Can't pickle <function worker at 0x5556de78daf0>: attribute lookup worker on __main__ failed
Am I missing something?我错过了什么吗?
The docs for memory_profiler: https://pypi.org/project/memory-profiler/ say the following if you use the decorator (@profile): memory_profiler 的文档: https://pypi.org/project/memory-profiler/如果您使用装饰器(@profile),请说明以下内容:
In this case the script can be run without specifying -m memory_profiler in the command line.在这种情况下,无需在命令行中指定 -m memory_profiler 即可运行脚本。
So I think you just need to run python MyScript.py
所以我认为你只需要运行python MyScript.py
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.