仅选择低于特定阈值的值

Question

Say I have a number of values: (left column is just the value count: 1,2,3, etc; right are the actual values)

1 5.2
2 1.43
3 3.54
4 887
5 0.35

What I want to do is reorder the values from decreasing to increasing (top-down), then I'd like to have python go through the values and keep picking the values (to be later used as an output) until it comes across a value that is at or above some threshold. For example:

5 0.35
2 1.43
3 3.54
1 5.2
4 887

Say, my threshold is at 5.0 so here I'd like the program to discard 1 and 4 (high values) and give 5, 2, and 3 as output along with their corresponding values. I hope that makes sense. Also as a trickier trick if (for whatever reason) my threshold only allows for 2 values I'd like it to ignore everything and give something like, 'No values found'.

The file they're located in from which I'll be pulling them (the values and counts) roughly looks like this:

  ID  some: value  another: value another: value another: value another: value another: value 1: 5.2

etc etc, each of the above mentioned values corresponds to a new line in the file. So the things I'm interested in would be located at row x, column 14 and 15 respectively.

The actual line would look like this:

Mod# 2 11494    Chi^2:  1.19608371367   Scale:  0.567691651772  Tin:    1499    Teff:   3400    Luminosity:     568.0   L   M-dot: 4.3497e-08   Tau: 2.44E-01   Dust composition: Fe    IRx1:   0.540471121182

I'm interested in IRx1 and the value following it.

Answer 1

Assuming your file has one number per line:

threshold = 5
with open('path/to/file') as infile:
    numbers = [float(line.strip()) for line in infile]
numbers.sort(reverse=True)
bigger = list(itertools.takewhile(lambda n: n<threshold, numbers))

If your file looks like this:

1 5.2
2 1.43
3 3.54
4 887
5 0.35

and you want your output to be set([2,3,5]) , then:

with open('path/to/file') as infile:
    numbers = dict([float(i) for i in line.strip()] for line in infile)
lines = sorted(numbers, key=numbers.__getitem__, reverse=True)
answer = list(itertools.takewhile(lambda n: numbers[n]<threshold, lines))

Given a file that looks like this:

Mod# 2 11494    Chi^2:  1.19608371367   Scale:  0.567691651772  Tin:    1499    Teff:   3400    Luminosity:     568.0   L   M-dot: 4.3497e-08   Tau: 2.44E-01   Dust composition: Fe    IRx1:   0.540471121182

where there is a tab ( \\t ) separating 11494 and Chi^2 , the following script should work:

def takeUntil(fpath, colname, threshold):
    lines = []
    with open(fpath) as infile:
        for line in infile:
            ldict = {}
            firsts = line.split('\t', 2)
            ldict[firsts[0] = float(firsts[1])
            splits = firsts[2].split('\t')
            ldict.update(dict(zip(firsts, itertools.islice(firsts, 1, len(firsts)))))
            lines.append(ldict)
    lines.sort(reverse=True, key=operator.itemgetter(colname))
    return [row['Mod#'] for row in itertools.takewhile(lambda row: row[colname]<threshold, lines)]

With that function, you should be able to specify which column's values you want to check to be under the threshold. Though this algorithm does have a higher space complexity (uses more RAM than absolutely necessary), you should be able to marshall/pickle lines after reading the file and continue from there for subsequent runs. This is especially useful if you have a huge input file which takes a while to process (as I suspect you might have)

Answer 2

The following solution assumes that the data was read in as a list of tuples.

Ex:

[(1,5.2),
(2,1.43),
(3,3.54),
(4,887),
(5,0.35)]

would be the list for the sample data in the problem.

def cutoff(threshold, data):
    sortedData = sorted(data, key=lambda x: x[1])
    finalList = filter(lambda x: x[1] < threshold, sortedData)
    return finalList if len(finalList) > 2 else 'No values found'

The first line of the function sorts the list by the values in the second place of the tuple.

The second line of the function then filters that resulting list so that only the elements in which the values are below the threshold remain.

The third line then returns the resulting sorted list if it contains more than two elements, and 'No values found' otherwise, which should accomplish what you're trying to do, less the file input.