[英]ResourceWarning: unclosed file <_io.BufferedReader name=4>
Consider the following program:考虑以下程序:
import tempfile
import unittest
import subprocess
def my_fn(f):
p = subprocess.Popen(['cat'], stdout=subprocess.PIPE, stdin=f)
yield p.stdout.readline()
p.kill()
p.wait()
def my_test():
with tempfile.TemporaryFile() as f:
l = list(my_fn(f))
class BuggyTestCase(unittest.TestCase):
def test_my_fn(self):
my_test()
my_test()
unittest.main()
Running it results in the following output:运行它会产生以下 output:
a.py:13: ResourceWarning: unclosed file <_io.BufferedReader name=4>
l = list(my_fn(f))
ResourceWarning: Enable tracemalloc to get the object allocation traceback
.
----------------------------------------------------------------------
Ran 1 test in 0.005s
OK
What is the actual cause of the warning and how to fix it?警告的实际原因是什么以及如何解决? Note that if I comment out
unittest.main()
the problem disappears, which means that it's specific to subprocess+unittest+tempfile.请注意,如果我注释掉
unittest.main()
问题就会消失,这意味着它特定于 subprocess+unittest+tempfile。
You should be closing the streams associated with the Popen()
object you opened.您应该关闭与您打开的 Popen
Popen()
object 关联的流。 For your example, that's the Popen.stdout
stream, created because you instructed the Popen()
object to create a pipe for the child process standard output. For your example, that's the
Popen.stdout
stream, created because you instructed the Popen Popen()
object to create a pipe for the child process standard output. The easiest way to do this is by using the Popen()
object as a context manager:最简单的方法是使用 Popen
Popen()
object 作为上下文管理器:
with subprocess.Popen(['cat'], stdout=subprocess.PIPE, stdin=f) as p:
yield p.stdout.readline()
p.kill()
I dropped the p.wait()
call as Popen.__exit__()
handles this for you, after having closed the handles.在关闭句柄后,我放弃了
p.wait()
调用,因为 Popen Popen.__exit__()
为您处理了这个问题。
If you want to further figure out exactly what cause the resource warning, then we can start by doing what the warning tells us, and enable the tracemalloc
module by setting the PYTHONTRACEMALLOC
environment variable :如果您想进一步找出导致资源警告的确切原因,那么我们可以从警告告诉我们的内容开始,并通过设置
PYTHONTRACEMALLOC
环境变量来启用tracemalloc
模块:
$ PYTHONTRACEMALLOC=1 python a.py
a.py:13: ResourceWarning: unclosed file <_io.BufferedReader name=4>
l = list(my_fn(f))
Object allocated at (most recent call last):
File "/.../lib/python3.8/subprocess.py", lineno 844
self.stdout = io.open(c2pread, 'rb', bufsize)
.
----------------------------------------------------------------------
Ran 1 test in 0.019s
OK
So the warning is thrown by a file opened by the subprocess
module.所以警告是由
subprocess
模块打开的文件引发的。 I'm using Python 3.8.0 here, and so line 844 in the trace points to these lines in subprocess.py
:我在这里使用 Python 3.8.0 ,因此跟踪中的第 844 行指向 subprocess.py 中的
subprocess.py
行:
if c2pread != -1:
self.stdout = io.open(c2pread, 'rb', bufsize)
c2pread
is the file handle for one half of a os.pipe()
pipe object created to handle communication from child process to Python parent process (created by the Popen._get_handles()
utility method , because you set stdout=PIPE
). c2pread
is the file handle for one half of a os.pipe()
pipe object created to handle communication from child process to Python parent process (created by the Popen._get_handles()
utility method , because you set stdout=PIPE
). io.open()
is exactly the same thing as the built-in open()
function . io.open()
与内置的open()
function 完全相同。 So this is where the BufferedIOReader
instance is created, to act as a wrapper for a pipe to receive the output the child process produces.所以这是创建
BufferedIOReader
实例的地方,它充当 pipe 的包装器,以接收子进程生成的 output。
You could also explicitly close p.stdout
:您还可以显式关闭
p.stdout
:
p = subprocess.Popen(['cat'], stdout=subprocess.PIPE, stdin=f)
yield p.stdout.readline()
p.stdout.close()
p.kill()
p.wait()
or use p.stdout
as a context manager:或使用
p.stdout
作为上下文管理器:
p = subprocess.Popen(['cat'], stdout=subprocess.PIPE, stdin=f)
with p.stdout:
yield p.stdout.readline()
p.kill()
p.wait()
but it's easier to just always use subprocess.Popen()
as a context manager, as it'll continue to work properly regardless of how many stdin
, stdout
or stderr
pipes you created.但是始终使用
subprocess.Popen()
作为上下文管理器会更容易,因为无论您创建了多少stdin
、 stdout
或stderr
管道,它都会继续正常工作。
Note that most subprocess
code examples don't do this, because they tend to use Popen.communicate()
, which closes the file handles for you.请注意,大多数
subprocess
代码示例不这样做,因为它们倾向于使用Popen.communicate()
,它会为您关闭文件句柄。
I also got some ResourceWarning: unclosed file <_io.FileIO name=7 mode='wb' closefd=True>
messages when running unit tests.运行单元测试时,我还收到了一些
ResourceWarning: unclosed file <_io.FileIO name=7 mode='wb' closefd=True>
消息。 I couldn't use the Popen
object as a context manager, so I did this in my classes __del__
method:我不能使用
Popen
object 作为上下文管理器,所以我在我的类__del__
方法中这样做了:
import subprocess
class MyClass:
def __init__(self):
self.child= subprocess.Popen(['dir'],
stdout=subprocess.PIPE,
stdin=subprocess.PIPE,
stderr=subprocess.PIPE)
def __del__(self):
self.child.terminate()
self.child.communicate()
You can just change my_fn
to use with
, then you won't need kill
and wait
.您只需将
my_fn
更改为使用with
,就不需要kill
和wait
。 Python'll close it for you: Python 会为你关闭它:
def my_fn(f):
with subprocess.Popen(['cat'], stdout=subprocess.PIPE, stdin=f) as p:
yield p.stdout.readline()
It fixed the issue in the way as follows:它以如下方式解决了这个问题:
def my_test():
with tempfile.TemporaryFile() as f:
x = my_fn(f)
l = list()
l.append(x)
# print(l)
Or the way as follows:或者如下方式:
def my_test():
with tempfile.TemporaryFile() as f:
l = list([my_fn(f)])
The following is helpful on the issue:以下对这个问题有帮助:
list Found at: builtins
list() -> new empty list
list(iterable) -> new list initialized from iterable's
items
https://docs.python.org/3/library/stdtypes.html#list mentioned the follows: https://docs.python.org/3/library/stdtypes.html#list提到了以下内容:
class list([iterable])
Lists may be constructed in several ways:
* Using a pair of square brackets to denote the empty list: []
* Using square brackets, separating items with commas: [a], [a, b, c]
* Using a list comprehension: [x for x in iterable]
* Using the type constructor: list() or list(iterable)
The constructor builds a list whose items are the same and in the same order as
iterable’s items. iterable may be either a sequence, a container that supports
iteration, or an iterator object. If iterable is already a list, a copy is made and
returned, similar to iterable[:]. For example, list('abc') returns ['a', 'b', 'c']
and list( (1, 2, 3) ) returns [1, 2, 3]. If no argument is given, the constructor
creates a new empty list, [].
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.