写入文件时，python行会自行连接

Question

I'm using python to generate training and testing data for 10-fold cross-validations, and to write the datasets to 2x10 separated files (each fold writes a training file and a testing file). And the weird thing is that when writing data to a file, there always is a line "missing". Actually, it might not even be "missing", since I discovered later that some line (only one line) in the middle of the file gets to concatenate itself to its previous line. So an output file should be something like the following (there should be 39150 lines in total):

44 1 90 0 44 0 45 46 0 1
55 -3 95 0 44 22 40 51 12 4
50 -3 81 0 50 0 31 32 0 1
44 -4 76 0 42 -30 32 34 2 1

However, I keep getting 39149 lines, and somewhere in the middle of the file seems to mess up like this:

44 1 90 0 44 0 45 46 0 1
55 -3 95 0 44 22 40 51 12 450 -3 81 0 50 0 31 32 0 1
44 -4 76 0 42 -30 32 34 2 1

My code:

def k_fold(myfile, myseed=1, k=10):
    # Load data
    data = open(myfile).readlines()

    # Shuffle input
    random.seed = myseed
    random.shuffle(data)

    # Compute partition size given input k
    len_total = len(data)
    len_part = int(math.floor(len_total / float(k)))

    # Create one partition per fold
    train = {}
    test = {}
    for i in range(k):
        test[i] = data[i * len_part:(i + 1) * len_part]
        train[i] = data[0:i * len_part] + data[(i + 1) * len_part:len_total]

    return train, test

if __name__ == "__main__":
    path = '....'  #some path and input
    input = '...'

    # Generate data
    [train, test] = k_fold(input)

    # Write data to files
    for i in range(10):
        train_old = path + 'tmp_train_' + str(i)
        test_old = path + 'tmp_test_' + str(i)

        trainF = open(train_old, 'a')
        testF = open(test_old, 'a')

        print(len(train[i]))

The strange thing is that I'm doing the same thing for the training and the testing dataset. The testing dataset outputs the correct file (4350 lines), but the training dataset has the above problem. I'm sure that the function returns the 39150 lines of training data, so I think the problem should be in the file writing part. Any body has any ideas how I could possibly done wrong? Thanks in advance!

Answer 1

I assume that the first half of the double length line is the last line of the original file.

The lines returned by readlines (or by iterating over the file) will all still end with the LF character '\\n' except the last line if the file doesn't end with an empty line. In that case, the shuffling that you do will put that '\\n' -less line somewhere in the middle of 'data' .

Either append an empty line to your original file or strip all lines before processing and add the newline to each line when writing back to a file.