Image saving with PIL throws corruption error for large file size

Question

I'm converting a large numpy array to tif (2.28G) with PIL, but I am getting a file corrupted error. I ran the exact same code with a smaller array before and it worked (1.7G). How can I go about and solve this problem?

edits:

• The numpy array I cannot convert has dtype ('uint8') and dimension (854715, 890, 3)
• The numpy array I can save is 'uint8' with dimensions (674775, 890, 3)
• I have 1960MB inactive memory and 40MB free memory

The code:

import sys
from PIL import Image
import numpy as np
import os

folder =  'Core4_4712_4803'

# read all file names from defined folder core_images into a 
Python list (collection of data)
images = os.listdir(folder) 
# list to collect UV stacked numpy array
combine_core_UV = []

print(images)
for img_name in images:
    im = Image.open(folder+"/"+img_name, 'r')
    img = np.array(im)

    x = 330
    x2 = 1663
    x3 = 3240
    x4 = 4457
    x5 = 5830
    y = 1675
    w = 890 # x2 - x1
    h = 8997 # y2 - y1
    crop_img1 = img[y:y+h, x:x+w]
    crop_img2 = img[y:y+h, x2:x2+w]
    crop_img3 = img[y:y+h, x3:x3+w]
    crop_img4 = img[y:y+h, x4:x4+w]
    crop_img5 = img[y:y+h, x5:x5+w]
    combined_image =np.concatenate((crop_img1,crop_img2,crop_img3,crop_img4,crop_img5),axis=0)
    combine_core_UV.append(combined_image)
    
   "concatenate pieces vertically"
combined_image_UV = np.concatenate(combine_core_UV,axis=0)
# save image
im = Image.fromarray(combined_image_UV)
im.save("uv_stacked_image_"+folder+".tif")

[picture error][1] [1]: https://i.stack.imgur.com/SW8JS.png

Answer 1

You appear to be loading all the parts you want into an enormous list which you then concatenate into an array which you then save as an image, which means you are trying to hold everything in a list and an array and an image at the same time thereby requiring 3x the RAM you might otherwise need.

Try creating your output image at the start, outside the loop then grab one piece at a time and paste it into the right place in the output image then move on to the next ROI without accumulating everything in memory except the output image.

You might also try adding compression='lzw' when saving your output file if you have libtiff installed.