I'm writing a Python script that takes in a (potentially large) file. Here is an example of a way that input file could be formatted:
class1 1:v1 2:v2 3:v3 4:v4 5:v5
class2 1:v6 4:v7 5:v8 6:v9
class1 3:v10 4:v11 5:v12 6:v13 8:v14
class2 1:v15 2:v16 3:v17 5:v18 7:v19
Where class1 and class2 are some number, eg 1 and -1. (A curious user may notice that this is a LIBSVM-related file, but knowing the software isn't necessary in this case.) The values v1, v2, ..., v19 represent any integer or float value. Obviously, my files would be much larger than this, in terms of total lines and length per line, which is why I'm concerned about efficiency here.
I am trying to check what is the greatest value to the left of a colon. In LIBSVM, these are called "features" and are always integers here. For instance, in the example I outlined above, line 1 has 5 as its largest feature. Line 2 has 6 as its largest feature, line 3 has 8 as its largest feature, and finally, line 4 has 7 as its largest feature. Since 8 is the largest of these values, that is my desired value. I'm looking at a file with possibly thousands of features per line, and many hundreds of thousands of lines .
The file satisfies the following properties:
Right now, my approach is to check each line, split up each line by a space, split up the final term by a colon, and then check the feature value. Following that, I do a procedure to check the maximum such featureNum.
file1 = open(...)
max = 0
for line in file1:
linesplit = line.rstrip('\n').split(' ')
val = linesplit[len(linesplit) - 1]
valsplit = val.split(':')
featureNum = valsplit[0]
if (featureNum > max):
max = featureNum
print max
file1.close()
But I'm hoping there is a better or more efficient way of doing this , eg some way of analyzing the file by only getting those terms that directly precede a newline character (maybe to avoid reading all the lines?). I'm new to Python so it wouldn't surprise me if I missed something obvious.
Possible reference: http://docs.python.org/library/stdtypes.html
Since you don't care about all the features in a line but just the last one, you don't need to split the whole line. I don't know if this is actually faster though, you need to time it and see. It definitely isn't as Pythonic as splitting the entire line.
def last_feature(line):
start = line.rfind(' ') + 1
end = line.rfind(':')
return int(line[start:end])
with open(...) as file1:
largest = max(last_feature(line) for line in file1)
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.