Running multiple commands in bash and creating new directories and moving csv files into the new directory

Question

I have several pcap files on which I would like to apply the same commands on (these are Argus commands a network flow tool)

• Convert the PCAP file to an argus file
• Run four Argus commands that outputs 4 .csv files
• Create a new directory based on the .pcap file name but with only the first part of the pcap name eg amazonecho_merge I didnt want the merge part as part of the new directory name

• Place all new .csv files into the new directory

    for file in *.pcap do argus -r *.pcap -w packet.argus #Run argus to get the flow volumn (totalbytes) and the flow duration (seconds) ra -r packet.argus -s bytes dur > flow_vol_dur.csv #Run argus to get the source and destination ports, merge both columns together and count how many occurances racluster -r packet.argus -n -s sport dport > ports.csv ra -r packet.argus -s stime ltime sport dport - dst port 53 > DNS.csv ra -r packet.argus -s stime ltime sport dport - dst port 123 > NTP.csv dir=$(echo ${file} | awk -F. '{print $1}' OFS=.) mkdir $dir #Move all newly created .csv files to the new directory mv $file*.csv $dir done 

I think naming the new directory is incorrect as I only want part of the name of the pcap file

I am sure there is a better way of running the commands especially this one

  ra -r packet.argus -s stime ltime sport dport - dst port 53 >   DNS.csv 
  ra -r packet.argus -s stime ltime sport dport - dst port 123 >   NTP.csv

when there is only slight change in the command I would like to know if there is a easier format of running these commands

Is there a way in bash to merge the columns from the different csv files into single csv file

e.g 

      file1.csv

      A,B

      file2.csv

      C,D

      Desired output.csv

      A,B,C,D

I have tried join and does not work is there any other bash command that will work?

Answer 1

It looks like most of your commands don't need to run once per file, so if you change the order you can save some runtime:

#!/usr/bin/env bash

argus -r *.pcap -w packet.argus

# Run argus to get the flow volumn (totalbytes) and the flow duration      (seconds)
ra -r packet.argus -s bytes dur > flow_vol_dur.csv

# Run argus to get the source and destination ports, merge both columns together and count how many occurances
racluster -r packet.argus -n -s sport dport > ports.csv
ra -r packet.argus -s stime ltime sport dport - dst port 53 > DNS.csv
ra -r packet.argus -s stime ltime sport dport - dst port 123 > NTP.csv

for file in *.pcap
    do
    dir=$(echo $file| awk -F_ '{print $1}')
    mkdir $dir
    # Move all newly created .csv files to the new directory 
    mv $file*.csv $dir
done