查找中位数的简单方法

Question

I have a data file and I perform few operations on the data. I can get solutions for all other operations just fine. I am not able to calculate the median only.

Input: Few lines from huge input.

00904bcabb02 00904bf7d758 676.0
0030657cc312 00904b1f1154 120.0
00306597852d 00904b48a3b6 572.0
00904b1f1154 00904bcabb02 120.0
00904b1f1154 00904bf7d758 120.0
00904b48a3b6 00904ba7a3eb 572.0
00022d1aa531 0006254f5810 2.0
00022dac729c 0006254f5810 2.0
00022dbd5c9e 0006254f5810 2.0
0006254f5810 0050dad80267 2.0
0006254f5810 00904be2b271 2.0
00022d097904 004096f41eb8 20.0
00022d2d30dd 004096f41eb8 20.0
004096f41eb8 00904b1e7852 20.0
00022d1406df 00022d36a6df 8.0
00022d36a6df 00022d8cb682 8.0
00022d36a6df 0030654a05fa 8.0
0004230dd7de 000423cbac29 33.0
0004231e4f43 000423cbac29 33.0
0030659b49f1 00904b310619 29.0

For every pair of col[0] col[1] I find the freq and the corresponding value's Average and Sum. I am trying to find the Median in set of pairtime . I am using numpy.median but that does not seem to be working. Any suggestion appreciated. Thanks

Code:

from collections import defaultdict
import numpy as np
paircount = defaultdict(int)
pairtime = defaultdict(float)
pairper = defaultdict(float)
timeavg = defaultdict(float)
timefreq = defaultdict(int)

#get number of pair occurrences and total time
with open('Input.txt', 'r') as f, open('Output.txt', 'w') as o:
    for numline, line in enumerate((line.split() for line in f), start=1):
        pair = line[0], line[1]
        paircount[pair] += 1
        pairtime[pair] += float(line[2])
    #timeavg = pairtime[pair]/paircount[pair]
    #pairper = dict((pair, c * 100.0 / numline) for (pair, c) in paircount.iteritems())
    for pair, freq in paircount.iteritems():
        timeavg = pairtime[pair] / freq
        med = np.median(np.pairtime[pair])
        #print pair[0], pair[1], c, pairper[pair], pairtime[pair]

        o.write("%s %s %s %.2f %.2f %s \n" % (pair[0], pair[1], freq, pairtime[pair], timeavg, med))
print 'done'

Error:

 Traceback (most recent call last):
  File "pair_one.py", line 20, in <module>
    med = np.median(np.pairtime[pair])
AttributeError: 'module' object has no attribute 'pairtime'

Answer 1

Your error is not really anything to do with the median, so this post should have a different title!

When Python says need more than 2 values to unpack , look at the line it's complaining about. Your iteration wants med, pair, freq - in other words it wants three values at a time, while what you're giving it is the result of iteritems() . iteritems() will always give you two values at a time since it always returns (key, val) pairs.

I think you just need to remove med, from your for-loop.

Answer 2

Your main problem is you're passing in a single floating point into the "median" function (pairtime[pair] contains the sum of the 3rd column values for the given c1,c2 pair). You need to pass the list of values instead. The way you calculate median is:

1) Take a list of numbers

2) Sort it

3) Pluck out the number in the exact center of the list. This is the median.

Here's my crack at a rewrite. I have not run it, so there may be syntax issues. But it should give you the general idea.

from collections import defaultdict
import numpy as np
paircount = defaultdict(int)
pairtime = defaultdict(float)
pairtimelist = defaultdict(list)
pairper = defaultdict(float)
timeavg = defaultdict(float)
timefreq = defaultdict(int)

#get number of pair occurrences and total time
with open('Input.txt', 'r') as f, open('Output.txt', 'w') as o:
    for numline, line in enumerate((line.split() for line in f), start=1):
        pair = line[0], line[1]
        paircount[pair] += 1
        pairtime[pair] += float(line[2])
        pairtimelist[pair].append(pairtime[pair])
    #timeavg = pairtime[pair]/paircount[pair]
    #pairper = dict((pair, c * 100.0 / numline) for (pair, c) in paircount.iteritems())
    for pair, freq in paircount.iteritems():
        timeavg = pairtime[pair] / freq
        med = np.median(pairtimelist[pair])
        #print pair[0], pair[1], c, pairper[pair], pairtime[pair]

        o.write("%s %s %s %.2f %.2f %s \n" % (pair[0], pair[1], freq, pairtime[pair], timeavg, med))
print 'done'

Answer 3

Median means a middle number of an array. Perhaps you mean this?

timelist=[]
for pair, freq in paircount.iteritems():
    timeavg = pairtime[pair] / freq
    parttimeArr=np.array(pairtime[pair])
    timelist.append(pairtime[pair])
timeArr=np.array(timelist)
median=np.median(timeArr)
print median

Answer 4

Replace:

med = np.median(np.pairtime[pair])

with:

med = np.median(pairtime[pair])

pairtime is a local variable, and not a numpy attribute.

EDIT

As @Fred S has pointed out, pairtime[pair] contains only the sum of the times, and not the complete series. I didn't notice it before. Since you will calculate many statistics from the time series, I believe a better approach would be to keep the whole time series instead of just the sum as @Fred S did in his answer. Then you can calculate all your statistics on the time series.

Here is a shot at a possible solution:

from collections import defaultdict
import numpy as np
pairtimelist = defaultdict(list)

with open('Input.txt', 'r') as f, open('Output.txt', 'w') as o:
    for numline, line in enumerate((line.split() for line in f), start=1):
        pair = line[0], line[1]
        pairtimelist[pair].append(float(line[2]))
    for pair in pairtimelist.iterkeys():
        timeavg = np.mean(pairtimelist[pair])
        timemed = np.median(pairtimelist[pair])
        timesum = np.sum(pairtimelist[pair])
        freq = len(pairtimelist[pair])

        o.write("%s %s %s %.2f %.2f %s \n" % (pair[0], pair[1], freq, timesum, timeavg, timemed))

Answer 5

The reason for the error is that you are prefixing pairtime with np , and NumPy has no idea of what pairtime is. If the intention is to convert pairtime to a NumPy array, you should write np.array(pairtime) . This should work, syntax wise:

from collections import defaultdict
import numpy as np
paircount = defaultdict(int)
pairtime = defaultdict(float)
pairper = defaultdict(float)
timeavg = defaultdict(float)
timefreq = defaultdict(int)

#get number of pair occurrences and total time
with open('Input.txt', 'r') as f, open('Output.txt', 'w') as o:
    for numline, line in enumerate((line.split() for line in f), start=1):
        pair = line[0], line[1]
        paircount[pair] += 1
        pairtime[pair] += float(line[2])
    #timeavg = pairtime[pair]/paircount[pair]
    #pairper = dict((pair, c * 100.0 / numline) for (pair, c) in paircount.iteritems())
    for pair, freq in paircount.iteritems():
        timeavg = pairtime[pair] / freq
        med = np.median(np.array(pairtime[pair]))
        # med = np.median(pairtime[pair]) # should work as well without np.array
        #print pair[0], pair[1], c, pairper[pair], pairtime[pair]

        o.write("%s %s %s %.2f %.2f %s \n" % (pair[0], pair[1], freq, pairtime[pair], timeavg, med))
print 'done'