简体繁体 English

存储大量实验和相应结果的方式

[英]Way to store large amount of experiments and corresponding results

原文 2011-03-04 19:09:22 8 1 database

I have an experiment setup that produces a few csv and text files every run. 我有一个实验设置，每次运行都会生成一些csv和文本文件。 In my research I run a number of these in parallel each day and trying to store and organized them in a directory heir-achy is getting rather inefficient and tedious. 在我的研究中，我每天并行运行许多此类文件，并试图将它们存储和组织到目录中，继承人效率低下且乏味。

One idea I thought of doing was using MySQL or something to store the the run description along with the associated output. 我想到的一个想法是使用MySQL或某种方式来存储运行说明以及相关的输出。 This however has the inefficiency in that I am storing a number of gigantic Strings in the table fields. 但是，这样做的效率很低，因为我在表字段中存储了许多巨大的字符串。

Basically I would like to know if a system exists that can automatically organize the output of runs. 基本上，我想知道是否存在可以自动组织运行输出的系统。 Such that it would give a list of the run descriptions, and upon choosing this you can request the corresponding outputs. 这样就可以给出运行说明的列表，选择后可以请求相应的输出。 This wouldn't have to be in any particular language as I can hopefully just add the step to a shell script. 不必使用任何特定的语言，因为我希望可以将步骤添加到Shell脚本中。

1 个解决方案

I assume that the csv and txt files are identically named for each run minus the extensions. 我假设每次运行减去扩展名后，csv和txt文件的名称都相同。 Have you considered a simple java or python script that could take all of the csv and txt files in a folder and put them into a single XML file where all of the information would be saved and with the data from the linked csv file and txt file together, meaning that for hundreds of file pairs (csv and txt) you could end up with a single XML file that could be opened to see the separate run results? 您是否考虑过一个简单的Java或python脚本，该脚本可以将一个文件夹中的所有csv和txt文件放入一个XML文件中，所有信息都将与该文件一起保存，其中包含链接的csv文件和txt文件中的数据在一起，这意味着对于成百上千个文件对（csv和txt），您可能最终得到一个可以打开以查看单独的运行结果的XML文件？