How can I write a list of lists into a txt file?

Question

I have a list of 16 elements, and each element is another 500 elements long. I would like to write this to a txt file so I no longer have to create the list from a simulation. How can I do this, and then access the list again?

Answer 1

Pickle will work, but the shortcoming is that it is a Python-specific binary format. Save as JSON for easy reading and re-use in other applications:

import json

LoL = [ range(5), list("ABCDE"), range(5) ]

with open('Jfile.txt','w') as myfile:
    json.dump(LoL,myfile)

The file now contains:

[[0, 1, 2, 3, 4], ["A", "B", "C", "D", "E"], [0, 1, 2, 3, 4]]

To get it back later:

with open('Jfile.txt','r') as infile:
    newList = json.load(infile)

print newList

Answer 2

To store it:

import cPickle

savefilePath = 'path/to/file'
with open(savefilePath, 'w') as savefile:
  cPickle.dump(myBigList, savefile)

To get it back:

import cPickle

savefilePath = 'path/to/file'
with open(savefilePath) as savefile:
  myBigList = cPickle.load(savefile)

Answer 3

Take a look pickle Object serialization. With pickle you can serialize your list and then save it to a text file. Later you can 'unpickle' the data from the text file. The data will be unpickled to a list and you can use it again in python. @inspectorG4dget beat me to the answer so take a look at.

Answer 4

While pickle is certainly a good option, for this particular question I would prefer simply saving it into a csv or just plain txt file with 16 columns using numpy .

import numpy as np

# here I use list of 3 lists as an example
nlist = 3

# generating fake data `listoflists`
listoflists = []
for i in xrange(3) :
    listoflists.append([i]*500)

# save it into a numpy array
outarr = np.vstack(listoflists)
# save it into a file
np.savetxt("test.dat", outarr.T)

Answer 5

I do recommend cPickle in this case, but you should take some "extra" steps:

• ZLIB the output.
• Encode or encrypt it.

By doing this you have these advantages:

• ZLIB will reduce its size.
• Encrypting may keep pickling hijacks off.

Yes, pickle is not safe! See this.