按行拆分文件并将第一个字符串保留为 output 文件的 header

Question

I have the file that look like that:

t_#_3_0 v_0_17  v_1_20  v_2_78  u_0_1   u_0_2   u_1_2
t_#_3_1 v_0_144 v_1_17  v_2_20  u_0_1   u_0_2   u_1_2
t_#_3_2 v_0_143 v_1_233 v_2_238 u_0_1   u_0_2   u_1_2
t_#_3_3 v_0_20  v_1_253 v_2_275 u_0_1   u_0_2   u_1_2
t_#_3_4 v_0_144 v_1_209 v_2_90  u_0_1   u_0_2   u_1_2
t_#_3_5 v_0_144 v_1_209 v_2_30  u_0_1   u_0_2   u_1_2
t_#_3_6 v_0_19  v_1_20  v_2_78  u_0_1   u_0_2   u_1_2
t_#_3_7 v_0_20  v_1_7   v_2_78  u_0_1   u_0_2   u_1_2
t_#_3_8 v_0_16  v_1_20  v_2_78  u_0_1   u_0_2   u_1_2
t_#_3_9 v_0_15  v_1_20  v_2_78  u_0_1   u_0_2   u_1_2

I would like to split it line by line and keep the first string as headers for my new files. My desired output should look like that.

file 1: t_#_3_0.txt (inside located line - t_#_3_0 v_0_17 v_1_20 v_2_78 u_0_1 u_0_2 u_1_2)
file 2: t_#_3_1.txt
file 3: t_#_3_2.txt

I tried split command, but it results in numerical indexes for output files.

I would appreciate your suggestions!

Thank you!

Olha

Answer 1

With GNU awk:

awk '{name=$1 ".txt"; print >name; close(name)}' file

Variable name contains content of first column with suffix .txt . print >name writes complete current row to file name .

Answer 2

My understanding of the requirements:

• each line in source file is to be copied to a new file
• new file is named after the contents of field (column) #1

One awk solution (assuming raw data is in file file.all ):

$ awk '{ fn=$1".txt" ; print > fn ; close (fn) } ' file.all
$ for f in t*#*txt
do
echo "+++++++++++++ $f"
cat $f
done

+++++++++++++ t_#_3_0.txt
t_#_3_0 v_0_17  v_1_20  v_2_78  u_0_1   u_0_2   u_1_2
+++++++++++++ t_#_3_1.txt
t_#_3_1 v_0_144 v_1_17  v_2_20  u_0_1   u_0_2   u_1_2
+++++++++++++ t_#_3_2.txt
t_#_3_2 v_0_143 v_1_233 v_2_238 u_0_1   u_0_2   u_1_2
+++++++++++++ t_#_3_3.txt
t_#_3_3 v_0_20  v_1_253 v_2_275 u_0_1   u_0_2   u_1_2
+++++++++++++ t_#_3_4.txt
t_#_3_4 v_0_144 v_1_209 v_2_90  u_0_1   u_0_2   u_1_2
+++++++++++++ t_#_3_5.txt
t_#_3_5 v_0_144 v_1_209 v_2_30  u_0_1   u_0_2   u_1_2
+++++++++++++ t_#_3_6.txt
t_#_3_6 v_0_19  v_1_20  v_2_78  u_0_1   u_0_2   u_1_2
+++++++++++++ t_#_3_7.txt
t_#_3_7 v_0_20  v_1_7   v_2_78  u_0_1   u_0_2   u_1_2
+++++++++++++ t_#_3_8.txt
t_#_3_8 v_0_16  v_1_20  v_2_78  u_0_1   u_0_2   u_1_2
+++++++++++++ t_#_3_9.txt
t_#_3_9 v_0_15  v_1_20  v_2_78  u_0_1   u_0_2   u_1_2

Answer 3

These results kinda surprised me. :)

sed

Just to pull something weird out of the toolbox as an example of why you should evaluate your methods now and then...

$: time sed -En 's/^([^ ]+)( .*)$/printf "%s%s\n" "\1" "\2" > \1.txt/e' file
real    0m0.859s
user    0m0.183s
sys     0m0.480s

I figured that would be slow, but it's nice to have in the toolbox just in case. Don't use pliers to drive nails.

awk

$: time awk '{ fn=$1".txt" ; print > fn ; close (fn) } ' file
real    0m0.141s
user    0m0.031s
sys     0m0.077s

Predictably faster, and likely a lot for a larger file.

But the one that surprised me -

Pure bash

$: time while read line; do echo "$line" > "${line%%[   ]*}"; done < file
real    0m0.015s
user    0m0.000s
sys     0m0.016s

NOTE - edited to use spaces AND/OR tabs

"${line%%[ ]*}" has both a space and a tab between the square brackets [ ] as field delimiters.

I think the time gain here is in the file management overhead. read is notoriously slow, but the OS is good at handling its own I/O, I suppose.

Maybe someone can chime in with a more in-depth analysis?

Elaborating the test out to a sample size of 10k records closes the gap between bash and awk a lot -

$: for x in {0..9999}; do echo "t_${x}_3_0 v_0_17  v_1_20  v_2_78  u_0_1   u_0_2   u_1_2"; done >| file

$: time while read line; do echo "$line" > "${line%% *}"; done < file
real    0m24.022s
user    0m2.360s
sys     0m11.938s

$: time awk '{ fn=$1".txt" ; print > fn ; close (fn) } ' file
real    0m27.284s
user    0m1.312s
sys     0m12.656s

$: $: time sed -En 's/^([^ ]+)( .*)$/printf "%s%s\n" "\1" "\2" > \1.txt/e' file
real    13m28.503s
user    1m48.374s
sys     8m22.970s

I suspect a lot of the difference with a smaller dataset is awk 's compile time, maybe?

Testing on 100k to confirm -

$: wc -c file # >5.5MB
5788890 file

bash :

real    8m42.666s
user    0m28.671s
sys     2m34.781s

awk :

real    8m15.096s
user    0m15.546s
sys     2m35.421s

I'm really surprised at how little difference there is.
I assume it's because most of the time is file I/O operations.