how to write list of lists in python for large lists?

Question

I want to write a list of lists into a file. The list of lists looks like:

a = [[1,2,3], [4,5,6], [7,8,9]]

I wrote

with open("my_file.txt", 'w') as fp:
    fp.write('\n'.join('{} {} {}'.format(x[0],x[1],x[2]) for x in a if x is not None))

However, my list a now has lists inside it containing 19 values, a lot: It looks like:

a = [[1,2,3,...,20], [21, 22, ..., 39], ...]

I do not want to write 16 more {} symbols and to modify the format command in there by adding another 16 x[i] where i = 3, 18. Is there a smart way to do this?

In addition, is there an efficient way to do this? My list a will contain 10,000 lists of 19 values each.

Generally, are there any simple tricks to improve efficiency of writing such results into a file?

Thank you!

Answer 1

The easiest option is to just serialize it to JSON:

import json
a = [[1,2,3], [4,5,6], [7,8,9]]
with open("my_file.txt", 'w') as fp:
    json.dump(a, fp)

You can load it back like this:

import json
with open("my_file.txt") as fp:
    a = json.load(fp)
    print(a)

Alternatively, if you don't want to use JSON, you can convert your list to strings and then use join as you did previously:

a = [[1,2,3], [4,5,6], [7,8,9]]
for index, item in enumerate(a):
    a[index] = str(item)
with open("my_file.txt", 'w') as fp:
    fp.write('\n'.join(a))

This is not as clean as JSON, but it does the job.

Answer 2

Iterate over the list and write each element of the sublist using print() with the * unpacking operator:

a = [[1,2,3], [4,5,6], [7,8,9]]

with open('my_file.txt', 'w') as fp:
    for sublist in a:
        if sublist is not None:    # filter out None items
            print(*sublist, file=fp)

The contents of my_file.txt :

1 2 3
4 5 6
7 8 9

This will not print any None items to the file. If you actually intended to remove empty lists then you could just write:

if sublist:
    print(...)

Answer 3

is there an efficient way to do this?

I would use pickle

The pickle module implements binary protocols for serializing and de-serializing a Python object structure. “Pickling” is the process whereby a Python object hierarchy is converted into a byte stream, and “unpickling” is the inverse operation, whereby a byte stream (from a binary file or bytes-like object) is converted back into an object hierarchy.

import pickle

a = [[1, 2, 3], [4, 5, 6], [7, 8, 9]]
with open('my_list.pkl', 'wb') as f:
    pickle.dump(a, f)
with open('my_list.pkl', 'rb') as f:
    a_from_file = pickle.load(f)
    print(a_from_file)

output

[[1, 2, 3], [4, 5, 6], [7, 8, 9]]