I have received a text file generated by a json dump in python that looks like this:
[0.1,0.1,0.2,0.3]
[0.1,0.3,0.4,0.3]
[0.1,0.1,0.3,0.3]
[0.3,0.1,0.5,0.3]
.
.
.
[0.1,0.1,0.3,0.3]
[0.3,0.4,0.6,0.3]
and so on for a considerable amount of lines ~>10,000,000
I would like to figure out the quickest/most efficient way to read from the file and actually convert them into lists.
I have a program that has a for loop that runs a particular operation with lists:
for x in range(filelength):
for y in list(each line from the file):
use the numbers from each list to perform certain operations
I was thinking of parsing out all the brackets from the text file and feeding each value comma separated into a blank list for each line (which would probably be slow and time consuming), but I thought there might be a feature of python to convert a list represented as a string easily into an actual list in python quickly.
Any thoughts or suggestions would be appreciated.
Use ast.literal_eval()
to parse each line back into a Python list:
import ast
with open(filename, 'r') as fh:
for line in fh:
listobj = ast.literal_eval(line)
ast.literal_eval()
takes a string and interprets it as Python literal values; lists and floating point values are directly supported:
>>> ast.literal_eval('[0.1,0.1,0.2,0.3]\n')
[0.1, 0.1, 0.2, 0.3]
You say this was "generated by a json dump", and each line looks like valid JSON, so the right thing to do is to parse each line as JSON:
import json
with open(filename) as f:
the_lists = map(json.loads, f)
Since you just want to iterate directly over the lists, it might be simpler to do the loads
right in your loop:
import json
with open(filename) as f:
for line in f:
for column in json.loads(line):
# your code here
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.