将数组中的相等元素分组

Question

I'm writing a program in python, which needs to sort through four columns of data in a text file, and return the four numbers the row with largest number in the third column for each set of identical numbers in the first column.

For example:

I need:

1.0     19.3    15.5    0.1
1.0     25.0    25.0    0.1
2.0     4.8     3.1     0.1
2.0     7.1     6.4     0.1
2.0     8.6     9.7     0.1
2.0     11.0    14.2    0.1
2.0     13.5    19.0    0.1
2.0     16.0    22.1    0.1
2.0     19.3    22.7    0.1
2.0     25.0    21.7    0.1
3.0     2.5     2.7     0.1
3.0     3.5     4.8     0.1
3.0     4.8     10.0    0.1
3.0     7.1     18.4    0.1
3.0     8.6     21.4    0.1
3.0     11.0    22.4    0.1
3.0     19.3    15.9    0.1
4.0     4.8     16.5    0.1
4.0     7.1     13.9    0.1
4.0     8.6     11.3    0.1
4.0     11.0    9.3     0.1
4.0     19.3    5.3     0.1
4.0     2.5     12.8    0.1
3.0     25.0    13.2    0.1

To return:

1.0     19.3    15.5    0.1
2.0     19.3    22.7    0.1
3.0     11.0    22.4    0.1
4.0     4.8     16.5    0.1

Here, the row [1.0, 19.3, 15.5, 0.1] is returned because 15.5 is the greatest third column value that any of the rows has, out of all the rows where 1.0 is the first number. For each set of identical numbers in the first column, the function must return the rows with the greatest value in the third column.

I am struggling with actually doing this in python, because the loop iterates over EVERY row and finds a maximum, not each 'set' of first column numbers.

Is there something about for loops that I don't know which could help me do this?

Below is what I have so far.

import numpy as np

C0,C1,C2,C3 = np.loadtxt("FILE.txt",dtype={'names': ('C0', 'C1', 'C2','C3'),'formats': ('f4', 'f4', 'f4','f4')},unpack=True,usecols=(0,1,2,3))

def FUNCTION(C_0,C_1,C_2,C_3):
    for i in range(len(C_1)):
        a = []
        a.append(C_0 [i])   
            for j in range(len(C_0)):
                if C_0[j] == C_0[i]:
                    a.append(C_0 [j])
        return a


print FUNCTION(C0,C1,C2,C3)

where C0,C1,C2, and C3 are columns in the text file, loaded as 1-D arrays. Right now I'm just trying to isolate the indexes of the rows with equal C0 values.

Answer 1

An approach could be to use a dict where the value is the row keyed by the first column item. This way you won't have to load the whole text file in memory at once. You can scan line by line and update the dict as you go.

Answer 2

I got some complex because of first and second rows... I believe 25.0 at (2, 3) is your mistake.

My code is not a mathematical solution, but it can be work.

import collections

with open("INPUT.txt", "r") as datasheet:
    data = datasheet.read().splitlines()

dataset = collections.OrderedDict()

for dataitem in data:
    temp = dataitem.split("    ")
    # I just wrote this code, input and output was seperated by four spaces
    print(temp)
    if temp[0] in dataset.keys():
        if float(dataset[temp[0]][1]) < float(temp[2]):
            dataset[temp[0]] = [temp[1], temp[2], temp[3]]
    else:
        dataset[temp[0]] = [temp[1], temp[2], temp[3]]

# Some sort code here

with open("OUTPUT.txt", "w") as outputsheet:
    for datakey in dataset.keys():
        datavalue = dataset[datakey]
        outputsheet.write("%s    %s    %s    %s\n" % (datakey, datavalue[0], datavalue[1], datavalue[2]))

Answer 3

Using Numpy and Lambda

Using the properties of a dict with some lambda functions does the trick..

data = np.loadtxt("FILE.txt",dtype={'names': ('a', 'b', 'c','d'),'formats': ('f4', 'f4', 'f4','f4')},usecols=(0,1,2,3))

# ordering by columns 1 and 3
sorted_data = sorted(data, key=lambda x: (x[0],x[2]))

# dict comprehension mapping the value of first column to a row
# this will overwrite all previous entries as mapping is 1-to-1
ret = {d[0]: list(d) for d in sorted_data}.values()

Alternatively, you can make it a (ugly) one liner..

ret = {
    d[0]: list(d)
    for d in sorted(np.loadtxt("FILE.txt",dtype={'names': ('a', 'b', 'c','d'),
                                                 'formats': ('f4', 'f4', 'f4','f4')},
                                          usecols=(0,1,2,3)),
                    key=lambda x: (x[0],x[2]))
}.values()

As @Fallen pointed out, this is an inefficient method as you need to read in the whole file. However, for the purposes of this example where the data set is quite small, it's reasonably acceptable.

Reading one line at a time

The more efficient way is reading in one line at a time.

import re

# Get the data
with open('data', 'r') as f:
    str_data = f.readlines()

# Convert to dict
d = {}
for s in str_data:
    data = [float(n) for n in re.split(r'\s+', s.strip())]

    if data[0] in d:
        if data[2] >= d[data[0]][2]:
            d[data[0]] = data
    else:
        d[data[0]] = data

print d.values()

The caveat here is that there's no other sorting metric so if you initially have a row for 1.0 with [1.0, 2.0, 3.0, 5.0] then any subsequent line with a 1.0 where the 3rd column is greater or equal to 3.0 will be overwritten, eg [1.0, 1.0, 3.0, 1.0]