从文本文件的数据中取平均值

Question

I have a text file as follows where there are two columns in between strings:

1   23
2   29
3   21
4   18
5   19
6   18
7   19
8   24
Cluster analysis done for this configuration!

1   23
2   22
3   19
4   18
5   23
6   17
7   19
8   31
9   21
10   27
11   19
Cluster analysis done for this configuration!

1   22
2   26
3   27
4   23
5   25
6   32
7   23
8   19
9   19
10   18
11   30
12   21
13   23
14   16
Cluster analysis done for this configuration!

1   23
2   19
3   23
4   27
5   20
6   17
7   15
8   22
9   16
10   23
11   20
12   23
Cluster analysis done for this configuration!

The desired output would be:

1 22.75
2 24.0
3 22.5
4 21.5
5 21.75
6 21.0
7 19.0
8 24.0
9 18.666666666666668
10 22.666666666666668
11 23.0
12 22.0
13 23.0
14 16.0

I would like to get an average for each of the numbers in the first column. If I take this example, the average value that corresponds to '1' would be: (23+23+22+23)/4 = 22.75 and so on for '2', '3'… Please note that the total numbers of rows are not the same in between the strings 'Cluster analysis….'but that's ok. For example, the average value for '14' would just be 16 in this case as there are no other numbers correspond to '14' except in '3rd' block.

I was thinking along the line that somehow one needs to print all the numbers between the strings 'Cluster analysis….' and then maybe a store in an array or so and then just do an average but couldn't implement it in code. Could anyone give me a lead?

I don't have any preference for the coding language; it just needs to solve the problem. I was thinking along with bash/shell but python is also welcome.

Answer 1

awk '/^[0-9]+ +[0-9]+$/ { # pick only lines with two numbers
         arr[$1] += $2    # accumulate the numbers in indexed bins
         n[$1]++          # keep track of how may numbers are in each bin
     }
     END {                     # finally,
         for (e in arr)        # for each bin
             print arr[e]/n[e] # divide
     }' your_input_file

Answer 2

Here is a solution, assuming the data are contained in a string called 's'

from collections import defaultdict

s = '1 23' #....etc

def list_struct():
    return list((int(), int()))

data = defaultdict(list_struct) # format: {id: [occurrences, total]}

for line in s.split('\n'):
    if line[0:1].isdigit(): # i'm assuming that all the lines that start with a number are the 'right' lines
        n, value = line.split()
        data[int(n)][0] += 1
        data[int(n)][1] += int(value)

for elem in data:
    print(elem, data[elem][1] / data[elem][0])

Output with your data:

1 22.75
2 24.0
3 22.5
4 21.5
5 21.75
6 21.0
7 19.0
8 24.0
9 18.666666666666668
10 22.666666666666668
11 23.0
12 22.0
13 23.0
14 16.0

Edit:

To read from a file just change the for loop to:

with open('f.txt', 'r') as f:
    for line in f:
        if line[0:1].isdigit(): # i'm assuming that all the lines that start with a number are the 'right' lines
            n, value = line.split()
            data[int(n)][0] += 1
            data[int(n)][1] += int(value)

Answer 3

A gimmick with bash because the question originally had a bash tag.

#!/bin/bash

div ()  # Arguments: dividend and divisor
{
  if [ $2 -eq 0 ]; then echo division by 0; exit; fi
  local p=15                            # precision
  local c=${c:-0}                       # precision counter
  local d=.                             # decimal separator
  local r=$(($1/$2)); echo -n $r        # result of division
  local m=$(($r*$2))
  [ $c -eq 0 ] && [ $m -ne $1 ] && echo -n $d
  [ $1 -eq $m ] || [ $c -eq $p ] && echo && return
  local e=$(($1-$m))
  c=$(($c+1))
  div $(($e*10)) $2
}

while read -r num val; do
  if [[ $num =~ ^[0-9] ]]; then
    a[$num]=$((a[$num]+$val))
    ((v[$num]++))
  fi
done < file

for((i=1; i<=${#a[@]}; i++)); do
  div ${a[$i]} ${v[$i]}
done

I used div function from there .

Output:

22.75
24
22.5
21.5
21.75
21
19
24
18.666666666666666
22.666666666666666
23
22
23
16

Answer 4

GNU datamash is a very handy tool for doing stats on groups of columnar data in scripts and one-liners. The catches here are having to remove the non-data lines first, and sorting the input numerically to get output in the desired order.

$ sed '/^$/d; /Cluster/d' input.txt | sort -k1,1n | datamash -Wg1 mean 2
1       22.75
2       24
3       22.5
4       21.5
5       21.75
6       21
7       19
8       24
9       18.666666666667
10      22.666666666667
11      23
12      22
13      23
14      16

Answer 5

Here's a simple python implementation.

# I'll store the output in a list of dicts
def get_averages(filename):
  results = []
  temp_dict = {'count':0, 'total':0, 'avg':0} # unnecessary; illustrate format
  for line in open(filename):
    words = line.rstrip('\n').split(' ')
    try: # if first word is a number
      if (int(words[0]) == 1): # interpret as start of new data list
        temp_dict = {'count':0, 'total':0, 'avg':0} # reset dict
      temp_dict['count'] += 1
      temp_dict['total'] += float(words[1])
    except ValueError:
      if (words[0] == 'Cluster'): # interpret as end of list
        temp_dict['avg']=temp_dict['total']/temp_dict['count']
        results.append(temp_dict)
      temp_dict = None # just to be safe
  return results

And then to get the desired form of your output:

results=get_averages(filename)
for i in range(len(results)):
  print('{} {}'.format(i,results[i]['avg']))