使用sed处理带分隔符的文本文件

Question

I have a ";" delimited file:

aa;;;;aa
rgg;;;;fdg
aff;sfg;;;fasg
sfaf;sdfas;;;           
ASFGF;;;;fasg
QFA;DSGS;;DSFAG;fagf

I'd like to process it replacing the missing value with a \\N . The result should be:

aa;\N;\N;\N;aa
rgg;\N;\N;\N;fdg
aff;sfg;\N;\N;fasg
sfaf;sdfas;\N;\N;\N         
ASFGF;\N;\N;\N;fasg
QFA;DSGS;\N;DSFAG;fagf

I'm trying to do it with a sed script:

sed "s/;\(;\)/;\\N\1/g" file1.txt  >file2.txt

But what I get is

aa;\N;;\N;aa
rgg;\N;;\N;fdg
aff;sfg;\N;;fasg
sfaf;sdfas;\N;;         
ASFGF;\N;;\N;fasg
QFA;DSGS;\N;DSFAG;fagf

Answer 1

You don't need to enclose the second semicolon in parentheses just to use it as \\1 in the replacement string. You can use ; in the replacement string:

sed 's/;;/;\\N;/g'

As you noticed, when it finds a pair of semicolons it replaces it with the desired string then skips over it, not reading the second semicolon again and this makes it insert \\N after every two semicolons.

A solution is to use positive lookaheads; the regex is /;(?=;)/ but sed doesn't support them.

But it's possible to solve the problem using sed in a simple manner: duplicate the search command; the first command replaces the odd appearances of ;; with ;\\N , the second one takes care of the even appearances. The final result is the one you need.

The command is as simple as:

sed 's/;;/;\\N;/g;s/;;/;\\N;/g'

It duplicates the previous command and uses the ; between g and s to separe them. Alternatively you can use the -e command line option once for each search expression:

sed -e 's/;;/;\\N;/g' -e 's/;;/;\\N;/g'

---

Update:

The OP asks in a comment "What if my file have 100 columns?"

Let's try and see if it works:

$ echo "0;1;;2;;;3;;;;4;;;;;5;;;;;;6;;;;;;;" | sed 's/;;/;\\N;/g;s/;;/;\\N;/g'
0;1;\N;2;\N;\N;3;\N;\N;\N;4;\N;\N;\N;\N;5;\N;\N;\N;\N;\N;6;\N;\N;\N;\N;\N;\N;

Look, ma! It works! :-)

---

Update #2

I ignored the fact that the question doesn't ask to replace ;; with something else but to replace the empty/missing values in a file that uses ; to separate the columns. Accordingly, my expression doesn't fix the missing value when it occurs at the beginning or at the end of the line.

As the OP kindly added in a comment, the complete sed command is:

sed 's/;;/;\\N;/g;s/;;/;\\N;/g;s/^;/\\N;/g;s/;$/;\\N/g'

or (for readability):

sed -e 's/;;/;\\N;/g;' -e 's/;;/;\\N;/g;' -e 's/^;/\\N;/g' -e 's/;$/;\\N/g'

The two additional steps replace ';' when they found it at beginning or at the end of line.

Answer 2

You can use this sed command with 2 s (substitute) commands:

sed 's/;;/;\\N;/g; s/;;/;\\N;/g;' file
aa;\N;\N;\N;aa
rgg;\N;\N;\N;fdg
aff;sfg;\N;\N;fasg
sfaf;sdfas;\N;\N;
ASFGF;\N;\N;\N;fasg
QFA;DSGS;\N;DSFAG;fagf

Or using lookarounds regex in a perl command:

perl -pe 's/(?<=;)(?=;)/\\N/g' file
aa;\N;\N;\N;aa
rgg;\N;\N;\N;fdg
aff;sfg;\N;\N;fasg
sfaf;sdfas;\N;\N;
ASFGF;\N;\N;\N;fasg
QFA;DSGS;\N;DSFAG;fagf

Answer 3

The main problem is that you can't use several times the same characters for a single replacement:

s/;;/..../g : The second ; can't be reused for the next match in a string like ;;;

If you want to do it with sed without to use a Perl-like regex mode, you can use a loop with the conditional command t :

sed ':a;s/;;/;\\N;/g;ta;' file

:a defines a label "a", ta go to this label only if something has been replaced.

For the ; at the end of the line (and to deal with eventual trailing whitespaces):

sed ':a;s/;;/;\\N;/g;ta; s/;[ \t\r]*$/;\\N/1' file

Answer 4

this awk one-liner will give you what you want:

awk -F';' -v OFS=';' '{for(i=1;i<=NF;i++)if($i=="")$i="\\N"}7' file

if you really want the line: sfaf;sdfas;\\N;\\N;\\N , this line works for you:

awk -F';' -v OFS=';' '{for(i=1;i<=NF;i++)if($i=="")$i="\\N";sub(/;$/,";\\N")}7' file

Answer 5

sed 's/;/;\\N/g;s/;\\N\([^;]\)/;\1/g;s/;[[:blank:]]*$/;\\N/' YourFile

• non recursive, onliner, posix compliant

Concept:

• change all ;
• put back unmatched one
• add the special case of last ; with eventually space before the end of line

Answer 6

This might work for you (GNU sed):

sed -r ':;s/^(;)|(;);|(;)$/\2\3\\N\1\2/g;t' file

There are 4 senarios in which an empty field may occur: at the start of a record, between 2 field delimiters, an empty field following an empty field and at the end of a record. Alternation can be employed to cater for senarios 1,2 and 4 and senario 3 can be catered for by a second pass using a loop ( :;...;t ). Multiple senarios can be replaced in both passes using the g flag.