Im trying to encode about 25000 images into a pickle file using the following code
f = open("dataset_encoding.pickle", "wb")
f.write(pickle.dumps(sampleDataset))
f.close()
Python consumes a lot of time executing this function on my PC, is there a verbose logging option for pickle.dumps()
so I can know that it is still running?
I couldn't find anything here
As mentioned in the comments, I was trying to pickle a list of length 4. To reduce the memory usage per pickle file, I decided to save my list into 4 different pickles. Following is the code I used to print a ProgressBar while the pickling completes:
import pickle
def printProgressBar (iteration, total, prefix = '', suffix = '', decimals = 1, length = 100, fill = '█', printEnd = "\r"):
"""
Call in a loop to create terminal progress bar
@params:
iteration - Required : current iteration (Int)
total - Required : total iterations (Int)
prefix - Optional : prefix string (Str)
suffix - Optional : suffix string (Str)
decimals - Optional : positive number of decimals in percent complete (Int)
length - Optional : character length of bar (Int)
fill - Optional : bar fill character (Str)
printEnd - Optional : end character (e.g. "\r", "\r\n") (Str)
"""
percent = ("{0:." + str(decimals) + "f}").format(100 * (iteration / float(total)))
filledLength = int(length * iteration // total)
bar = fill * filledLength + '-' * (length - filledLength)
print('\r%s |%s| %s%% %s' % (prefix, bar, percent, suffix), end = printEnd)
if iteration == total:
print()
def verbosePickle(data_list, filename, prefix_text):
print("sampleDataset["+str(index)+"] Length :"+str(len(sample)))
with open(filename, "wb") as f:
pickle.dump(len(data_list), f)
printProgressBar(0, len(data_list)-1, prefix = prefix_text, suffix = "Complete", length = 50)
for data_index, data in enumerate(data_list):
printProgressBar(data_index, len(data_list)-1, prefix = prefix_text, suffix = "Complete", length = 50)
pickle.dump(data, f)
for index, sample in enumerate(sampleDataset):
print("Saving Pickle Files")
verbosePickle(sample, "sampleDataset"+index+".pickle", "Saving pickle "+index+" :")
Standard Output:
Saving pickle 0: |█████████████████████-----------------------------| 44.0% Complete
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.