Parse multiple lines of a large file in python, store them in lists

Question

The python code below:

pkts=rdpcap("abcFile.pcap",100)

def parsePcap():
    for pkt in pkts:
        if IP in pkt:
            ip_src=pkt[IP].src
            ip_dst=pkt[IP].dst
            ip_proto=pkt[IP].proto
        yield(ip_src,ip_dst,ip_proto)

with open("filenameTEST", 'w') as f:
    for i in parsePcap():
        f.write("%s,%s,%d\n" % i)

And saves response in the file filenameTEST:

121.14.142.72,0.32.59.21,6
123.152.135.217,0.3.17.121,17
71.229.65.158,0.48.101.12,17
58.20.154.23,0.191.51.126,17
68.249.101.222,0.62.29.118,6

I want to store first two values (strings in one list) and last integer in a separate list:

attribute = []
nodePairs = []
with open("filenameTEST") as f:
    while(True):
        myArr = [b.split(',') for b in f.readline().split()]
        for i in myArr:
            attribute.append((i[0],i[1],i[2])) 
            nodePairs.append((i[0],i[1]))

The last segment of the code is incorrect, but what i need is two lists that have attribute as the list with all 3 values and nodepairs as the first two values from each line. Also, i[2] is an integer.

Answer 1

try using csv module it would be easier to solve this problem.

Don't read all rows into a list .

Process your rows as you produce them. If you need to filter the data first, use a generator function:

import csv

def getstuff(filename, criterion):
    with open(filename, "rb") as csvfile:
        datareader = csv.reader(csvfile)
        count = 0
        for row in datareader:
                yield row

You now only hold one row in memory, instead of your thousands of lines.

yield makes a function a generator function , which means it won't do any work until you start looping over it.

this could also be helpful: http://lethain.com/handling-very-large-csv-and-xml-files-in-python/