[英]Parse multiple lines of a large file in python, store them in lists
The python code below: 下面的python代码:
pkts=rdpcap("abcFile.pcap",100)
def parsePcap():
for pkt in pkts:
if IP in pkt:
ip_src=pkt[IP].src
ip_dst=pkt[IP].dst
ip_proto=pkt[IP].proto
yield(ip_src,ip_dst,ip_proto)
with open("filenameTEST", 'w') as f:
for i in parsePcap():
f.write("%s,%s,%d\n" % i)
And saves response in the file filenameTEST: 并在文件filenameTEST中保存响应:
121.14.142.72,0.32.59.21,6
123.152.135.217,0.3.17.121,17
71.229.65.158,0.48.101.12,17
58.20.154.23,0.191.51.126,17
68.249.101.222,0.62.29.118,6
I want to store first two values (strings in one list) and last integer in a separate list: 我想将前两个值(一个列表中的字符串)和最后一个整数存储在一个单独的列表中:
attribute = []
nodePairs = []
with open("filenameTEST") as f:
while(True):
myArr = [b.split(',') for b in f.readline().split()]
for i in myArr:
attribute.append((i[0],i[1],i[2]))
nodePairs.append((i[0],i[1]))
The last segment of the code is incorrect, but what i need is two lists that have attribute as the list with all 3 values and nodepairs as the first two values from each line. 代码的最后一段是不正确的,但我需要的是两个列表,其中属性为列表,所有3个值和nodepairs作为每行的前两个值。 Also, i[2]
is an integer. 另外, i[2]
是整数。
try using csv module it would be easier to solve this problem. 尝试使用csv模块,这将更容易解决这个问题。
Don't read all rows into a list . 不要将所有行读入列表 。
Process your rows as you produce them. 在生成行时处理行。 If you need to filter the data first, use a generator function: 如果需要先过滤数据,请使用生成器函数:
import csv
def getstuff(filename, criterion):
with open(filename, "rb") as csvfile:
datareader = csv.reader(csvfile)
count = 0
for row in datareader:
yield row
You now only hold one row in memory, instead of your thousands of lines. 您现在只在内存中保留一行 ,而不是数千行。
yield
makes a function a generator function , which means it won't do any work until you start looping over it. yield
使函数成为生成函数 ,这意味着在开始循环之前它不会做任何工作。
this could also be helpful: http://lethain.com/handling-very-large-csv-and-xml-files-in-python/ 这也可能有所帮助: http : //lethain.com/handling-very-large-csv-and-xml-files-in-python/
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.