[英]Very large python function definition in exec() call crashes Django but does not crash directly executed Python code
I have a very large (~400k lines) Python function that I am attempting to define through an exec()
call.我有一个非常大的(~400k 行) Python function 我试图通过
exec()
调用来定义。 If I run the following Python script:如果我运行以下 Python 脚本:
exec("""def blah()
# 400k lines of IF/THEN/ELSE
""", globals())
blah()
By calling Python from the command line, it works fine.通过从命令行调用 Python ,它工作正常。
However, if I do the same within a Django instance, it crashes the server without any error message or stack trace, which I can only assume is due to a segmentation fault.但是,如果我在 Django 实例中执行相同操作,它会在没有任何错误消息或堆栈跟踪的情况下使服务器崩溃,我只能假设这是由于分段错误。
Both Django runserver and the above script are run from the same Conda enviroment, and both have unlimited stack available (confirmed by printing out resource.getrlimit
in Django). Django runserver 和上面的脚本都是在相同的 Conda 环境中运行的,并且都有无限的可用堆栈(通过在 Django 中打印出
resource.getrlimit
来确认)。
Here's my full ulimit -a
output:这是我的完整
ulimit -a
output:
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 515017
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) unlimited
cpu time (seconds, -t) unlimited
max user processes (-u) 4096
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
The command sequence to launch the Django server is as follows:启动 Django 服务器的命令序列如下:
source activate <conda env name>
python manage.py runserver
This is the shell input/output leading to the crash:这是导致崩溃的 shell 输入/输出:
(faf) [pymaster@t9dpyths3 faf]$ python manage.py runserver 9000
Watching for file changes with StatReloader
Performing system checks...
System check identified no issues (0 silenced).
August 04, 2020 - 08:25:19
Django version 3.0.3, using settings 'faf.settings'
Starting development server at http://127.0.0.1:9000/
Quit the server with CONTROL-C.
[04/Aug/2020 08:25:25] "GET /projects/ HTTP/1.1" 200 13847
[04/Aug/2020 08:26:49] "PUT /projects/projectname/ HTTP/1.1" 200 76 # This event triggers the exec
(faf) [pymaster@t9dpyths3 faf]$
The problem might be due to int(s), float(s) and others may cause segmentation fault该问题可能是由于int(s)、float(s) 和其他可能导致分段错误
Please try setting the environmental flag PYTHONMALLOC=debug
请尝试设置环境标志
PYTHONMALLOC=debug
This might allow your code to run without running into segmentation errors, if you do still get an error you should be able to catch it using.这可能允许您的代码运行而不会遇到分段错误,如果您仍然遇到错误,您应该能够使用它来捕获它。
PYTHONMALLOC=debug python3 -X tracemalloc=10
You might also want to check out: faulthandler您可能还想查看: faulthandler
This module contains functions to dump Python tracebacks explicitly, on a fault, after a timeout, or on a user signal.
该模块包含在发生故障、超时或用户信号时显式转储 Python 回溯的功能。 Call faulthandler.enable() to install fault handlers for the SIGSEGV, SIGFPE, SIGABRT, SIGBUS, and SIGILL signals.
调用 faulthandler.enable() 为 SIGSEGV、SIGFPE、SIGABRT、SIGBUS 和 SIGILL 信号安装故障处理程序。 You can also enable them at startup by setting the PYTHONFAULTHANDLER environment variable or by using the -X faulthandler command line option.
您还可以通过设置 PYTHONFAULTHANDLER 环境变量或使用 -X faulthandler 命令行选项在启动时启用它们。
Adding this for more clarity since it's related;添加它是为了更清楚,因为它是相关的; the following is taken from the answer provided by Darrrrrren and is a tweak to make faulthandler run on threaded django applications:
以下内容来自 Darrrrrren 提供的答案,是使故障处理程序在线程 django 应用程序上运行的调整:
So I was able to get a stack trace by initializing Python with faulthandler, but additionally I had to run
manage.py runserver
--nothreading --noreload
- for some reason if you do not disable threading with Django, even faulthandler will not print a stack trace.因此,我可以通过使用故障处理程序初始化 Python 来获得堆栈跟踪,但另外我必须运行
manage.py runserver
--nothreading --noreload
- 由于某种原因,如果您不使用 Django 禁用线程,即使故障处理程序也不会打印堆栈跟踪。
This sounds like a job to divide and conquer !这听起来像是分而治之的工作!
Split your exec block up into parts to find where it fails, attempting to catch BaseException
rather than Exception
and dumping progress将您的 exec 块拆分为多个部分以查找失败的位置,尝试捕获
BaseException
而不是Exception
并转储进度
If you believe you're hitting a segfault, you can handle it using signal.signal(signalnum, handler)
example如果你认为你遇到了段错误,你可以使用
signal.signal(signalnum, handler)
例子来处理它
As they're guaranteed to be a contained block of logic, you could begin new blocks to execute by splitting at def
and if
statements.由于它们被保证是一个包含的逻辑块,因此您可以通过在
def
和if
语句处拆分来开始执行新块。 If most if
statements are at the highest scope, you should be able to split directly on them, otherwise some additional scope detection will be needed.如果大多数
if
语句都在最高 scope,您应该能够直接对其进行拆分,否则将需要一些额外的 scope 检测。
import signal
import sys
CONTENT_AND_POS = {
"text_lines": [], # first attempt is exec("") without if
"block_line_no": 1, # first block should be at line 1+
}
def report(text_lines, line_no, msg=""):
""" display progress to the console """
print("running code block at {}:{}\n{}".format(
line_no, msg, text_lines)) # NOTE reordered from args
def signal_handler_segfault(signum, frame):
""" try to show where the segfault occurred """
report(
"\n".join(CONTENT_AND_POS["text_lines"]),
CONTENT_AND_POS["block_line_no"],
"SIGNAL {}".format(signum)
)
sys.exit("caught segfault")
# initial setup
signal.signal(signal.SIGSEGV, signal_handler_segfault)
path_code_to_exec = sys.argv[1] # consider argparse
print("reading from {}".format(path_code_to_exec))
# main entrypoint
with open(path_code_to_exec) as fh:
for line_no, line in enumerate(fh, 1): # files are iterable by-line
if line.startswith(("def", "if")): # new block to try
text_exec_block = "\n".join(CONTENT_AND_POS["text_lines"])
try:
exec(text_exec_block, globals())
except BaseException as ex:
report(
text_exec_block,
CONTENT_AND_POS["block_line_no"],
str(repr(ex)))
# catching BaseException will squash exit, ctrl+C, et al.
sys.exit("caught BaseException")
# reset for the next block
CONTENT_AND_POS["block_line_no"] = line_no # new block begins
CONTENT_AND_POS["text_lines"].clear()
# continue with new or existing block
CONTENT_AND_POS["text_lines"].append(line)
# execute the last block (which is otherwise missed)
exec_text_lines(
CONTENT_AND_POS["text_lines"],
CONTENT_AND_POS["block_line_no"]
)
print("successfully executed {} lines".format(line_no))
If this still ends silently, output the line number of each block before executing it.如果这仍然静默结束,output 执行之前每个块的行号。 You may need to write to a file or
sys.stdout/stderr
to ensure output isn't lost您可能需要写入文件或
sys.stdout/stderr
以确保 output 不会丢失
If you're using Python 2 (perhaps accidentally), you're simply passing too much to exec
如果您使用的是
exec
2 (可能是意外),那么您只是传递了太多执行
You can reproduce this as follows (also see a relevant codegolf !)您可以按如下方式重现此内容(另请参阅相关的 codegolf !)
% python2
>>> exec(
... """if True:
... pass
... """ * (200 * 1000) # 400k lines
... )
segmentation fault python2
You should be able to fix this by breaking it up (described in my other answer ), or by writing the code to a file and importing it instead (as suggested/already implemented in comments )您应该能够通过分解它(在我的其他答案中描述)或通过将代码写入文件并导入它来解决此问题(如建议/已在评论中实现)
This limit on exec
should be fixed in Python 3 (RecursionError) , but may affect a few unlucky versions (see ticket).这个
exec
限制应该在 Python 3 (RecursionError) 中修复,但可能会影响一些不幸的版本(见票)。
So I was able to get a stack trace by initializing Python with faulthandler
, but additionally I had to run manage.py runserver --nothreading --noreload
- for some reason if you do not disable threading with Django, even faulthandler will not print a stack trace.因此,我可以通过使用
faulthandler
初始化 Python 来获得堆栈跟踪,但另外我必须运行manage.py runserver --nothreading --noreload
- 出于某种原因,如果您不使用 Django 禁用线程,即使 faulthandler 也不会打印堆栈跟踪。
Fatal Python error: Segmentation fault
Current thread 0x00007fe61836b740 (most recent call first):
File "/apps/AADD/projects/FAF/Web App/faf/modelling/views.py", line 42 in index
File "/apps/AADD/envs/faf/lib/python3.6/site-packages/django/core/handlers/base.py", line 113 in _get_response
File "/apps/AADD/envs/faf/lib/python3.6/site-packages/django/core/handlers/exception.py", line 34 in inner
File "/apps/AADD/envs/faf/lib/python3.6/site-packages/django/utils/deprecation.py", line 94 in __call__
File "/apps/AADD/envs/faf/lib/python3.6/site-packages/django/core/handlers/exception.py", line 34 in inner
File "/apps/AADD/envs/faf/lib/python3.6/site-packages/django/utils/deprecation.py", line 94 in __call__
File "/apps/AADD/envs/faf/lib/python3.6/site-packages/django/core/handlers/exception.py", line 34 in inner
File "/apps/AADD/envs/faf/lib/python3.6/site-packages/django/utils/deprecation.py", line 94 in __call__
File "/apps/AADD/envs/faf/lib/python3.6/site-packages/django/core/handlers/exception.py", line 34 in inner
File "/apps/AADD/envs/faf/lib/python3.6/site-packages/django/utils/deprecation.py", line 94 in __call__
File "/apps/AADD/envs/faf/lib/python3.6/site-packages/django/core/handlers/exception.py", line 34 in inner
File "/apps/AADD/envs/faf/lib/python3.6/site-packages/django/utils/deprecation.py", line 94 in __call__
File "/apps/AADD/envs/faf/lib/python3.6/site-packages/django/core/handlers/exception.py", line 34 in inner
File "/apps/AADD/envs/faf/lib/python3.6/site-packages/django/utils/deprecation.py", line 94 in __call__
File "/apps/AADD/envs/faf/lib/python3.6/site-packages/django/core/handlers/exception.py", line 34 in inner
File "/apps/AADD/envs/faf/lib/python3.6/site-packages/django/utils/deprecation.py", line 94 in __call__
File "/apps/AADD/envs/faf/lib/python3.6/site-packages/django/core/handlers/exception.py", line 34 in inner
File "/apps/AADD/envs/faf/lib/python3.6/site-packages/django/core/handlers/base.py", line 75 in get_response
File "/apps/AADD/envs/faf/lib/python3.6/site-packages/django/core/handlers/wsgi.py", line 133 in __call__
File "/apps/AADD/envs/faf/lib/python3.6/site-packages/django/contrib/staticfiles/handlers.py", line 68 in __call__
File "/apps/AADD/envs/faf/lib/python3.6/wsgiref/handlers.py", line 137 in run
File "/apps/AADD/envs/faf/lib/python3.6/site-packages/django/core/servers/basehttp.py", line 197 in handle_one_request
File "/apps/AADD/envs/faf/lib/python3.6/site-packages/django/core/servers/basehttp.py", line 172 in handle
File "/apps/AADD/envs/faf/lib/python3.6/socketserver.py", line 724 in __init__
File "/apps/AADD/envs/faf/lib/python3.6/socketserver.py", line 364 in finish_request
File "/apps/AADD/envs/faf/lib/python3.6/socketserver.py", line 351 in process_request
File "/apps/AADD/envs/faf/lib/python3.6/socketserver.py", line 320 in _handle_request_noblock
File "/apps/AADD/envs/faf/lib/python3.6/socketserver.py", line 241 in serve_forever
File "/apps/AADD/envs/faf/lib/python3.6/site-packages/django/core/servers/basehttp.py", line 216 in run
File "/apps/AADD/envs/faf/lib/python3.6/site-packages/django/core/management/commands/runserver.py", line 139 in inner_run
File "/apps/AADD/envs/faf/lib/python3.6/site-packages/django/core/management/commands/runserver.py", line 104 in run
File "/apps/AADD/envs/faf/lib/python3.6/site-packages/django/core/management/commands/runserver.py", line 95 in handle
File "/apps/AADD/envs/faf/lib/python3.6/site-packages/django/core/management/base.py", line 369 in execute
File "/apps/AADD/envs/faf/lib/python3.6/site-packages/django/core/management/commands/runserver.py", line 60 in execute
File "/apps/AADD/envs/faf/lib/python3.6/site-packages/django/core/management/base.py", line 328 in run_from_argv
File "/apps/AADD/envs/faf/lib/python3.6/site-packages/django/core/management/__init__.py", line 395 in execute
File "/apps/AADD/envs/faf/lib/python3.6/site-packages/django/core/management/__init__.py", line 401 in execute_from_command_line
File "manage.py", line 17 in main
File "manage.py", line 21 in <module>
Segmentation fault
If I provide unlimited stack space, the exec()
actually works in Django, but only with --nothreading
.如果我提供无限的堆栈空间,则
exec()
实际上在 Django 中有效,但仅适用于--nothreading
。 So I have a hunch that Django is somehow restricting stack size to spawned off threads.所以我有一种预感,Django 以某种方式将堆栈大小限制为产生的线程。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.