拆分一个大的txt文件以执行grep-UNIX

Question

I work (unix, shell scripts) with txt files that are millions field separate by pipe and not separated by \\n or \\r . something like this:

field1a|field2a|field3a|field4a|field5a|field6a|[...]|field1d|field2d|field3d|field4d|field5d|field6d|[...]|field1m|field2m|field3m|field4m|field5m|field6m|[...]|field1z|field2z|field3z|field4z|field5z|field6z|

All text is in the same line.

The number of fields is fixed for every file.

(in this example I have field1=name; field2=surname; field3=mobile phone; field4=email; field5=office phone; field6=skype )

When I need to find a field (ex field2 ), command like grep doesn't work (in the same line).

I think that a good solution can be do a script that split every 6 field with a "\\n" and after do a grep. I'm right? Thank you very much!

Answer 1

With awk :

$ cat a
field1a|field2a|field3a|field4a|field5a|field6a|field1d|field2d|field3d|field4d|field5d|field6d|field1m|field2m|field3m|field4m|field5m|field6m|field1z|field2z|field3z|field4z|field5z|field6z|



$ awk -F"|" '{for (i=1;i<NF;i=i+6) {for (j=0; j<6; j++) printf $(i+j)"|"; printf "\n"}}' a

field1a|field2a|field3a|field4a|field5a|field6a|
field1d|field2d|field3d|field4d|field5d|field6d|
field1m|field2m|field3m|field4m|field5m|field6m|
field1z|field2z|field3z|field4z|field5z|field6z|

Here you can easily set the length of line.

Hope this helps !

Answer 2

you can use sed to split the line in multiple lines:

 sed 's/\(\([^|]*|\)\{6\}\)/\1\n/g' input.txt > output.txt

explanation:

• we have to use heavy backslash-escaping of (){} which makes the code slightly unreadable.

• but in short:

  • the term (([^|]*|){6}) (backslashes removed for readability) between s/ and /\\1 , will match:

    • [^|]* any character but '|', repeated multiple times

    • | followed by a '|'

    • the above is obviously one column and it is grouped together with enclosing parantheses ( and )

    • the entire group is repeated 6 times {6}

    • and this is again grouped together with enclosing parantheses ( and ) , to form one full set

the rest of the term is easy to read:

• replace the above (the entire dataset of 6 fields) with \\1\\n , the part between / and /g

• \\1 refers to the "first" group in the sed-expression (the "first" group that is started, so it's the entire dataset of 6 fields)

• \\n is the newline character

• so replace the entire dataset of 6 fields by itself followed by a newline

• and do so repeatedly (the trailing g )

Answer 3

you can use sed to convert every 6th | to a newline.

In my version of tcsh I can do:

sed 's/\(\([^|]\+|\)\{6\}\)/\1\n/g' filename

consider this:

> cat bla
a1|b2|c3|d4|

> sed 's/\(\([^|]\+|\)\{6\}\)/\1\n/g' bla
a1|b2|
c3|d4|

This is how the regex works:

• [^|] is any non- | character.
• [^|]\\+ is a sequence of at least one non- | characters.
• [^|]\\+| is a sequence of at least one non- | characters followed by a | .
• \\([^|]\\+|\\) is a sequence of at least one non- | characters followed by a | , grouped together
• \\([^|]\\+|\\)\\{6\\} is 6 consecutive such groups.
• \\(\\([^|]\\+|\\)\\{6\\}\\) is 6 consecutive such groups, grouped together.

The replacement just takes this sequence of 6 groups and adds a newline to the end.

Answer 4

Here is how I would do it with awk

awk -v RS="|" '{printf $0 (NR%7?RS:"\n")}' file
field1a|field2a|field3a|field4a|field5a|field6a|[...]
field1d|field2d|field3d|field4d|field5d|field6d|[...]
field1m|field2m|field3m|field4m|field5m|field6m|[...]
field1z|field2z|field3z|field4z|field5z|field6z|

Just adjust the NR%7 to number of field you to what suites you.

Answer 5

What about printing the lines on blocks of six?

$ awk 'BEGIN{FS=OFS="|"} {for (i=1; i<=NF; i+=6) {print $(i), $(i+1), $(i+2), $(i+3), $(i+4), $(i+5)}}' file
field1a|field2a|field3a|field4a|field5a|field6a
field1d|field2d|field3d|field4d|field5d|field6d
field1m|field2m|field3m|field4m|field5m|field6m
field1z|field2z|field3z|field4z|field5z|field6z

Explanation

• BEGIN{FS=OFS="|"} set input and output field separator as | .
• {for (i=1; i<=NF; i+=6) {print $(i), $(i+1), $(i+2), $(i+3), $(i+4), $(i+5)}} loop through items on blocks of 6. Every single time, print six of them. As print end up writing a new line, then you are done.

Answer 6

If you want to treat the files as being in multiple lines, then make \\n the field separator. For example, to get the 2nd column, just do:

tr \| \\n < input-file | sed -n 2p

To see which columns match a regex, do:

tr \| \\n < input-file | grep -n regex