如何计算多个csv文件中数字的平均值？

Question

I've files like the following as replicates from a simulation experiment I've been doing:

generation, ratio_of_player_A, ratio_of_player_B, ratio_of_player_C

So, the data is something like

0, 0.33, 0.33, 0.33

1, 0.40, 0.40, 0.20

2, 0.50, 0.40, 0.10

etc

Now, since I run this is in multiples, I've around ~1000 files for each experiment, giving various such numbers. Now, my problem is to average them all for 1 set of experiment.

Thus, I would like to have a file that contains the average ratio after each generation (averaged over multiple replicates, ie files)

All the replicate output files which need to be averaged are names like output1.csv, output2.csv, output3.csv .....output1000.csv

I'd be obliged if someone could help me out with a shell script, or a python script.

Answer 1

If I understood well, let's say you have 2 file like those:

$ cat file1
0, 0.33, 0.33, 0.33
1, 0.40, 0.40, 0.20
2, 0.50, 0.40, 0.10

$ cat file2
0, 0.99, 1, 0.02
1, 0.10, 0.90, 0.90
2, 0.30, 0.10, 0.30

And you want to do the mean between column of both files. So here is a way for the first column :

Edit : I found a better way, using pd.concat :

all_files = pd.concat([file1,file2]) # you can easily put your 1000 files here
result = {}
for i in range(3): # 3 being number of generations
    result[i] = all_files[i::3].mean()
result_df = pd.DataFrame(result)
result_df
                       0     1     2
ratio_of_player_A  0.660  0.25  0.40
ratio_of_player_B  0.665  0.65  0.25
ratio_of_player_C  0.175  0.55  0.20

---

Other way with merge, but one needs to perform multiple merges

import pandas as pd

In [1]: names = ["generation", "ratio_of_player_A", "ratio_of_player_B", "ratio_of_player_C"]
In [2]: file1 = pd.read_csv("file1", index_col=0, names=names)
In [3]: file2 = pd.read_csv("file2", index_col=0, names=names)
In [3]: file1
Out[3]:     
       ratio_of_player_A  ratio_of_player_B  ratio_of_player_C
generation                                                         
0                        0.33               0.33               0.33
1                        0.40               0.40               0.20
2                        0.50               0.40               0.10    

In [4]: file2
Out[4]: 
            ratio_of_player_A  ratio_of_player_B  ratio_of_player_C
generation                                                         
0                        0.99                1.0               0.02
1                        0.10                0.9               0.90
2                        0.30                0.1               0.30



In [5]: merged_file = file1.merge(file2, right_index=True, left_index=True, suffixes=["_1","_2"])
In [6]: merged_file.filter(regex="ratio_of_player_A_*").mean(axis=1)
Out[6]
generation
0             0.66
1             0.25
2             0.40
dtype: float64

Or this way (a bit faster I guess) :

merged_file.ix[:,::3].mean(axis=1) # player A

You can merge recursively before applying the mean() method if you have more than one file.

If I misunderstood the question, please show us what you expect from file1 and file2.

Ask if there is something you don't understand.

Hope this helps !

Answer 2

The following should work:

from numpy import genfromtxt

files = ["file1", "file2", ...]

data = genfromtxt(files[0], delimiter=',')
for f in files[1:]:
    data += genfromtxt(f, delimiter=',')

data /= len(files)

Answer 3

You can load each of the 1000 experiments in a dataframe, sum them all then calculate the mean.

filepath = tkinter.filedialog.askopenfilenames(filetypes=[('CSV','*.csv')]) #select your files
for file in filepath:
    df = pd.read_csv(file, sep=';', decimal=',')
    dfs.append(df)

temp = dfs[0] #creates a temporary variable to store the df
for i in range(1,len(dfs)): #starts from 1 cause 0 is stored in temp
    temp = temp + dfs[i];
result = temp/len(dfs)

Answer 4

your problem is not very clear.. if i understand it right..

>temp
for i in `ls *csv`
more "$i">>temp;

then you have all the data from different files in one big file. try to load in sqlite database (1. Create a table 2.Insert the data) after that you can query your data like. select sum(columns)/count(columns) from yourtablehavingtempdata etc. try to see sqlite since your data is tabular.sqlite will be better suited in my opinion.