Why is Python throwing a memory exception on a triple nested for?

Question

I am writing a demonstration of for loops that outputs all the possible colours in the RGB spectrum. The intention is that will help students understand how for loops work.

import csv

print("Started")

for_max = 256


with open('spectrum.csv', 'w', newline='') as csvfile:
    writer = csv.writer(csvfile)
    spectrum = []
    head = ["R", "G", "B", "Hex"]
    spectrum.append(head)
    for r in range(0, for_max):
        for g in range(0, for_max):
            for b in range(0, for_max):
                r_hex = format(r, '02x')
                g_hex = format(g, '02x')
                b_hex = format(b, '02x')
                hex_string = str("#") + str(r_hex) + str(g_hex) + str(b_hex)
                spectrum.append([format(r, '03'), format(g, '03'), format(b, '03'), hex_string])
    writer.writerows(spectrum)
print("Finished")

Unfortunately, I am currently getting a memory overflow.

Traceback (most recent call last): File "C:/[...]/rgb_for.py", line 31, in MemoryError

I have checked that the final list is smaller than the Python list maximum and indeed it is. What, therefore, could be causing this?

Answer 1

Building the list then dumping it into the CSV in its entirety could arguably be called bad practice. What if your program needs to output many rows but falls over half way through? Only outputting at the end would lead to loss of data. This method is also more computationally intensive, as dumping a huge list is quite some task, so it takes longer to execute.

The better method is to output each row as it is ready. Try this on for size;

import csv

print("Started")

for_max = 256

with open('spectrum.csv', 'w', newline='') as csvfile:
    writer = csv.writer(csvfile)
    out_list = []
    head = ["R", "G", "B", "Hex"]
    writer.writerow(head)
    for r in range(0, for_max):
        for g in range(0, for_max):
            for b in range(0, for_max):
                r_hex = format(r, '02x')
                g_hex = format(g, '02x')
                b_hex = format(b, '02x')
                hex_string = str("#") + str(r_hex) + str(g_hex) + str(b_hex)
                out_list = [format(r, '03'), format(g, '03'), format(b, '03'), hex_string]
                writer.writerow(out_list)
print("Finished")

An added joy of this method is that you get to watch the output filesize steadily increase!