如何将“n”个数组保存到“n”个npy文件？

Question

I have strings like these ones in a file:

tactccctccatcccataatataagacatggtcaaacttggcacggtcttcaaaactaatctttaacttttaatttctca
gttggtatcagaggagtggtcctcggaggtttccgcgacggcgccgtccaccaaaccttccgtgcagacgcgtctgctgc
tgttatggtgggtaaaacatatgaaaggcggcccaagcagcccataggccaggccacgccatgagatcaagattaaaggg
tgttcaggccagcaatggacagtttccgggcactgtgccggctccgcgcgcctcggtcccggctcgcgctcgcgcgctcg
cgttatcagcacgagctgcctaagctaatagatactaaaaaaaaatttctgataaaaaaccatatttattatcgactcat
aaagtggtatcagagcttgaagatcctaaagatggcgagtaacaatgttcccttccaagtcccggtgctcacaaagagca
tggtatcagacagccgatcacaagcttccgctccctaacccatcgccgacgacctcaccggccaccacatcctcttctcc
gctggaatagctcagttggttagagcgtgtggctgttaaccacaaggtcggaggttcaagccctccttctagcgcttttt
gagcactgctcccgtccagcaaacggtaccccaggtaccggtaccccggtacgaaacttaatctgaccattgaattagag
gtgttaacccagatcaaatgcctcgttccctgggccgcctcattgtgaggggaagtattgcgacaagtcactaactcttc
cttgagcagagagttgctcattggaatactacaatattcatatattttactggagcttctaaacggattccttcaccatg
aagtgtgatgattggaggtgtgcgggttcgagcgctgtacaactacaccggagaggagcctgatgagctctccttcaaat
gtccactgacctgtaatagagaactggattgctcatgagaccccgatcgttcacttgtaatacattgatagttttcttta
atcaacgggagatgtccaagaccgaataactaagcgaattagtttggacttatcagcgaaatttcagtgatgctgaaagc
aaaatcagtttatcaacgggagatgtccaagaccgaataactaagcgaattagtttggacttatcagcgaaatttcagtg
ctactatgtttaaaaatatgttgaacaaaaaattctctctgttaaaaaaaacagttgggaaatatttttcgaacaattaa
gtaatagccagtcgcacctctactgtactcgaaatacatgtcagtgagattaacattcattattctgaccagacggctta
ttggccatcaaatgagggaaacaggccagctcctcgacaaaaaaagtggcatgagatttatcaaaggattgagaaatgcc

I need to one-hot encode each line and save to a numpy binary file. I wrote this function for the task.

def ohe(padded_file):
    integer_encoder = LabelEncoder()  
    one_hot_encoder = OneHotEncoder(categories='auto')   
    input_features = []
    with open(padded_file, 'r') as padded:
        padded = padded.readlines()
        arr_nums = [i for i in range(len(padded))]
        for linha in padded:
            linha = linha.strip()
            integer_encoded = integer_encoder.fit_transform(list(linha))
            integer_encoded = np.array(integer_encoded).reshape(-1, 1)
            one_hot_encoded = one_hot_encoder.fit_transform(integer_encoded)
            stacked = np.stack(one_hot_encoded.toarray())
            input_features.append(stacked)
        np.savez_compressed("seq_arrays.npz", *input_features)

It works, but if I need to handle a bigger file containing many thousands of lines and longer strings, depending on the RAM capacity of the computer it can't do the job. So, for the latter case, I thougth I could write a numpy binary file for each line and then join in a same compressed file or even let them in a directory to be used when necessary. WRT that I did the following:

def ohe(padded_file):
    integer_encoder = LabelEncoder()  
    one_hot_encoder = OneHotEncoder(categories='auto')   
    with open(padded_file, 'r') as padded:
        padded = padded.readlines()
        arr_nums = [i for i in range(len(padded))]
        for linha in padded:
            linha = linha.strip()
            integer_encoded = integer_encoder.fit_transform(list(linha))
            integer_encoded = np.array(integer_encoded).reshape(-1, 1)
            one_hot_encoded = one_hot_encoder.fit_transform(integer_encoded)
            stacked = np.stack(one_hot_encoded.toarray())
            for num in arr_nums:
                save(f'arr_{num}.npy',stacked)

It creates a number of files based on the number of lines, but all of them store the same data, which is the last string one-hot encoded. I need to have the first file with the first line one-hot encoded, the second file with the second line one-hot encoded and so on. What am I doing wrong?

Answer 1

Use a construct that does not read the whole file but line by line

    with open(padded_file, 'r') as padded:
        for i,linha in enumerated(padded):
            linha = linha.strip()
            # ....other stuff
            save(f'arr_{i}.npy',stacked)