[英]How to read a CSV file from a stream and process each line as it is written?
I would like to read a CSV file from the standard input and process each row as it comes.我想从标准输入中读取 CSV 文件并处理每一行。 My CSV outputting code writes rows one by one, but my reader waits the stream to be terminated before iterating the rows.我的 CSV 输出代码一行一行地写入行,但我的读者在迭代行之前等待 stream 终止。 Is this a limitation of csv
module?这是csv
模块的限制吗? Am I doing something wrong?难道我做错了什么?
My reader code:我的阅读器代码:
import csv
import sys
import time
reader = csv.reader(sys.stdin)
for row in reader:
print "Read: (%s) %r" % (time.time(), row)
My writer code:我的作家代码:
import csv
import sys
import time
writer = csv.writer(sys.stdout)
for i in range(8):
writer.writerow(["R%d" % i, "$" * (i+1)])
sys.stdout.flush()
time.sleep(0.5)
Output of python test_writer.py | python test_reader.py
python 的python test_writer.py | python test_reader.py
python test_writer.py | python test_reader.py
: python test_writer.py | python test_reader.py
:
Read: (1309597426.3) ['R0', '$']
Read: (1309597426.3) ['R1', '$$']
Read: (1309597426.3) ['R2', '$$$']
Read: (1309597426.3) ['R3', '$$$$']
Read: (1309597426.3) ['R4', '$$$$$']
Read: (1309597426.3) ['R5', '$$$$$$']
Read: (1309597426.3) ['R6', '$$$$$$$']
Read: (1309597426.3) ['R7', '$$$$$$$$']
As you can see all print statements are executed at the same time, but I expect there to be a 500ms gap.如您所见,所有打印语句都是同时执行的,但我预计会有 500 毫秒的间隔。
As it says in the documentation ,正如文档中所说,
In order to make a
for
loop the most efficient way of looping over the lines of a file (a very common operation), thenext()
method uses a hidden read-ahead buffer.为了使for
循环成为循环文件行的最有效方式(一种非常常见的操作),next()
方法使用隐藏的预读缓冲区。
And you can see by looking at the implementation of the csv
module (line 784) that csv.reader
calls the next()
method of the underlyling iterator (via PyIter_Next
).通过查看csv
模块(第 784 行)的实现,您可以看到csv.reader
调用了底层迭代器的next()
方法(通过PyIter_Next
)。
So if you really want unbuffered reading of CSV files, you need to convert the file object (here sys.stdin
) into an iterator whose next()
method actually calls readline()
instead.因此,如果您真的想要无缓冲读取 CSV 文件,则需要将文件 object (此处为sys.stdin
)转换为迭代器,其next()
方法实际上调用readline()
。 This can easily be done using the two-argument form of the iter
function.这可以使用iter
器 function 的两个参数形式轻松完成。 So change the code in test_reader.py
to something like this:因此,将test_reader.py
中的代码更改为如下所示:
for row in csv.reader(iter(sys.stdin.readline, '')):
print("Read: ({}) {!r}".format(time.time(), row))
For example,例如,
$ python test_writer.py | python test_reader.py
Read: (1388776652.964925) ['R0', '$']
Read: (1388776653.466134) ['R1', '$$']
Read: (1388776653.967327) ['R2', '$$$']
Read: (1388776654.468532) ['R3', '$$$$']
[etc]
Can you explain why you need unbuffered reading of CSV files?您能解释一下为什么需要无缓冲读取 CSV 文件吗? There might be a better solution to whatever it is you are trying to do.无论您尝试做什么,都可能有更好的解决方案。
Maybe it's a limitation.也许这是一个限制。 Read this http://docs.python.org/using/cmdline.html#cmdoption-unittest-discover-u阅读此http://docs.python.org/using/cmdline.html#cmdoption-unittest-discover-u
Note that there is internal buffering in file.readlines() and File Objects (for line in sys.stdin) which is not influenced by this option.请注意,file.readlines() 和文件对象(用于 sys.stdin 中的行)中有内部缓冲,不受此选项的影响。 To work around this, you will want to use file.readline() inside a while 1: loop.要解决这个问题,您需要在 while 1: 循环中使用 file.readline()。
I modified test_reader.py as follows:我修改了 test_reader.py 如下:
import csv, sys, time
while True:
print "Read: (%s) %r" % (time.time(), sys.stdin.readline())
Output Output
python test_writer.py | python test_reader.py
Read: (1309600865.84) 'R0,$\r\n'
Read: (1309600865.84) 'R1,$$\r\n'
Read: (1309600866.34) 'R2,$$$\r\n'
Read: (1309600866.84) 'R3,$$$$\r\n'
Read: (1309600867.34) 'R4,$$$$$\r\n'
Read: (1309600867.84) 'R5,$$$$$$\r\n'
Read: (1309600868.34) 'R6,$$$$$$$\r\n'
Read: (1309600868.84) 'R7,$$$$$$$$\r\n'
You are flushing stdout, but not stdin.您正在刷新标准输出,但不是标准输入。
Sys.stdin
also has a flush()
method, try using that after each line read if you really want to disable the buffering. Sys.stdin
也有一个flush()
方法,如果您真的想禁用缓冲,请尝试在每行读取后使用该方法。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.