将目录文件中的特定列提取到新文件中

Question

I have a collection of 11064 files, and they all have same file extension ReadsPerGene.out.tab. They are in a directory. All the files have 556 lines, 4 columns.

Filenames look like this:
SRR123.ReadsPerGene.out.tab
SRR456.ReadsPerGene.out.tab
SRR555.ReadsPerGene.out.tab
DRR789.ReadsPerGene.out.tab
...

File looks like this:
for SRR123ReadsPerGene.out.tab        for SRR789.ReadsPerGene.out.tab
A    45   67   78                       A    89O  90   34
B    17   40   23                       B    129  96   45
C    27   50   19                       C     60  56   91
...  ...  ...  ...                     ...   ...  ...  ...                                           

First, I want to judge whether the first column of all the files are the same.

If it's true, I want to create an output.txt file with 665 lines, 11065 columns. The 1th column is the first column of every file(because they are same). And 2th of 11065th columns of output.txt are the 2th column of each input file, I want to add the specific filename as the first line for every column.

The output.txt looks like this:

      SRR123                SRR789              SRR456        ...
A        45                 89O                66            ...
B        17                 129                480           ...
C        27                  60                78            ...
...      ...               ...               ...             ...

The following are my answers. **
**1. get all filenames

#!/bin/bash
cd ~
filepath=/home/shared/maize/bam_rsem
cd ${filepath}
for file in $(ls *.ReadsPerGene.out.tab)
do
   echo $file >> ~/filename.txt
done

2. get all the first column in one file

#!/bin/bash
cd ~
OUT=result2.txt
touch $OUT
filepath=/home/shared/maize/bam_rsem/
for file in $(cat filename.txt)
do
   filePATH=`echo ${filepath}$file`
   cut -f 1 $filePATH | sed 1i\ ${file} >$OUT.tmp1
   paste $OUT $OUT.tmp1 >$OUT.tmp
   rm $OUT.tmp1
   mv $OUT.tmp $OUT
done

3. compare whether the first column is identical with other columns in result2.txt
I have no idea now.
4. create an output.txt

#!/bin/bash
cd ~
OUT=result2.txt
touch $OUT
filepath=/home/shared/maize/bam_rsem/
for file in $(cat filename.txt)
do
   filePATH=`echo ${filepath}$file`
   cut -f 1 $filePATH | sed 1i\ ${file} >$OUT.tmp1
   paste $OUT $OUT.tmp1 >$OUT.tmp
   rm $OUT.tmp1
   mv $OUT.tmp $OUT
done

cut -f 1 result2.txt >$OUT.tmp2
paste $OUT.tmp2 $OUT >$OUT.tmp3
rm $OUT.tmp2
mv $OUT.tmp3 $OUT

What should I do for my script? It is really slow to execute my script in Linux. Or should I write up a Python script to handle it,but I have never learned python or Perl and I just know a little about Linux.

I'm so sorry that my English is poor, I can not reply in time. Anyway, thanks for all your answers!

Answer 1

One in awk. The filenames to process are in files (due to large number of them):

$ cat files
SRR123.ReadsPerGene.out.tab
SRR789.ReadsPerGene.out.tab

The awk program is to run in the directory with the data files ( split ing the first . separated part of the filename for the header, ie. leading path would make the header name pretty lengthy):

$ awk '
BEGIN{OFS="\t"}
{
    files[NR]=$0                                # hash filenames from file files
}
END{
    for(i=1;i<=NR;i++) {                        # loop files
        nr=0
        split(files[i],t,".")
        h[nr]=h[nr] OFS t[1]                    # build header
        while((getline < files[i])>0) {         # using getline to read data records
            nr++                                # d[++nr] order not same in all awks
            d[nr]=d[nr] OFS $2                  # append data fields to previous
            if(i==1) {                          # get headers from first file
                h[(refnr=nr)]=$1
            } else if($1!=h[nr]) {              # check that they stay the same
                print "Nonmatching field name"
                exit                            # or exit without output
            }
        }
        if(nr!=refnr) {                         # also record count must be the same
            print "Nonmatching record count"
            exit
        }
        close(files[i])
    }
    for(i=0;i<=refnr;i++)                       # output part
        print h[i] d[i]
}' files

Output:

        SRR123  SRR789
A       45      89O
B       17      129
C       27      60
...     ...     ...

[++nr] order not same in all awks : Apparently some awks prefer d[++nr]=d[nr] OFS $2 and some d[nr]=d[++nr] OFS $2 so separate nr++ works for both.

Update :

If the files are in a different path and the filenames in the file files don't have paths included, replace intelligently:

split(files[i],t,".")
...
while((getline < files[i])>0) {

with

file="home/shared/maize/bam_rsem/" files[i]
split(file,t,".")
...
while((getline < file)>0) {

AND

close(files[i])

with

close(file)

Answer 2

Try this and let me know if it worked in the comments section of this answer.

import pandas as pd
import glob

files = sorted(glob.glob("*.log.out", recursive=False))

#dropped_col_1 = list()
#kept_col_2 = list()
drop_files = dict()
keep_files = dict()
ref_file_name = 'SRR123.log.out'
df_ref_file = pd.read_csv(ref_file_name, sep='\t', header=None)
for i, filename in enumerate(files):
    df_file = pd.read_csv(filename, sep='\t', header=None)
    if df_ref_file['0'] != df_file['0']:
        drop_files.update({filename: df_file['0'].tolist()})
        #dropped_col_1.append(df_file['0'].tolist())
    else:
        keep_files.update({filename: df_file['1'].tolist()})        
        #kept_col_2.append(df_file['1'].tolist())

df = pd.DataFrame(keep_files, index=df_ref_file['0'])
df.index.names = ['ID']
df.reset_index(inplace=True)
# check the shape of the dataframe
df.shape