使用sed替换引号中的文本

Question

I have this test file.

[root@localhost ~]# cat f.txt 
"a aa"  MM  "bbb  b"
MM    MM
MM"b b "
[root@localhost ~]#

I want to replace all space characters in the quotes, note, just in the quotes. All characters out of the quotes should not be touched. That is to say, what I want is something similar to:

"a_aa"  MM  "bbb__b"
MM    MM
MM"b_b_"

Can this be implemented using sed ?

Thanks,

Answer 1

This is an entirely non-trivial question.

This works replacing the first space inside quotes with underscore:

$ sed 's/\("[^ "]*\) \([^"]*"\)/\1_\2/g' f.txt
"a_aa"  MM  "bbb_ b"
MM    MM
MM"b_b "
$

For this example, where there are no more than two spaces inside any of the quotes, it is tempting to simply repeat the command, but it gives an incorrect result:

$ sed -e 's/\("[^ "]*\) \([^"]*"\)/\1_\2/g' \
>     -e 's/\("[^ "]*\) \([^"]*"\)/\1_\2/g' f.txt
"a_aa"_ MM  "bbb_ b"
MM    MM
MM"b_b_"
$

If your version of sed supports 'extended regular expressions', then this works for the sample data:

$ sed -E \
>    -e 's/^(([^"]*("[^ "]*")?)*)("[^ "]*) ([^"]*")/\1\4_\5/' \
>    -e 's/^(([^"]*("[^ "]*")?)*)("[^ "]*) ([^"]*")/\1\4_\5/' \
>    -e 's/^(([^"]*("[^ "]*")?)*)("[^ "]*) ([^"]*")/\1\4_\5/' \
>    f.txt
"a_aa"  MM  "bbb__b"
MM    MM
MM"b_b_"
$

You have to repeat that ghastly regex for every space within double quotes - hence three times for the first line of data.

The regex can be explained as:

• Starting at the beginning of a line,
• Look for sequences of 'zero or more non-quotes, optionally followed by a quote, no spaces or quotes, and a quote', the whole assembly repeated zero or more times,
• Followed by a quote, zero or more non-quotes, non-spaces, a space, and zero or more non-quotes, and a quote.
• Replace the matched material with the leading part, the material at the start of the current quoted passage, an underscore, and the trailing material of the current quoted passage.

Because of the start anchor, this has to be repeated once per blank...but sed has a looping construct, so we can do it with:

$ sed -E -e ':redo
>            s/^(([^"]*("[^ "]*")?)*)("[^ "]*) ([^"]*")/\1\4_\5/
>            t redo' f.txt
"a_aa"  MM  "bbb__b"
MM    MM
MM"b_b_"
$

The :redo defines a label; the s/// command is as before; the t redo command jumps to the label if there was any substitution done since the last read of a line or jump to a label.

---

Given the discussion in the comments, there are a couple of points worth mentioning:

1. The -E option applies to sed on MacOS X (tested 10.7.2). The corresponding option for the GNU version of sed is -r (or --regex-extended ). The -E option is consistent with grep -E (which also uses extended regular expressions). The 'classic Unix systems' do not support EREs with sed (Solaris 10, AIX 6, HP-UX 11).

2. You can replace the ? I used (which is the only character that forces the use of an ERE instead of a BRE) with * , and then deal with the parentheses (which require backslashes in front of them in a BRE to make them into capturing parentheses), leaving the script:

    sed -e ':redo s/^\\(\\([^"]*\\("[^ "]*"\\)*\\)*\\)\\("[^ "]*\\) \\([^"]*"\\)/\\1\\4_\\5/g t redo' f.txt 

   This produces the same output on the same input - I tried some slightly more complex patterns in the input:

    "a aa" MM "bbb b" MM MM MM"bb " "cc""dd""ee" X " f "" g " "CC" "DD" "EE" x " F " " G " 

   This gives the output:

    "a_aa" MM "bbb__b" MM MM MM"b_b_" "c_c""d_d""e__e" X "_f_""_g_" "C_C" "D_D" "E__E" x "_F_" "_G_" 

3. Even with BRE notation, sed supported the \\{0,1\\} notation to specify 0 or 1 occurrences of the previous RE term, so the ? version could be translated to a BRE using:

    sed -e ':redo s/^\\(\\([^"]*\\("[^ "]*"\\)\\{0,1\\}\\)*\\)\\("[^ "]*\\) \\([^"]*"\\)/\\1\\4_\\5/g t redo' f.txt 

   This produces the same output as the other alternatives.

Answer 2

A somehow unusual answer in XSLT 2.0 :

<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
    version="2.0">
    <xsl:output method="text"></xsl:output>
    <xsl:template name="init">
        <xsl:for-each select="tokenize(unparsed-text('f.txt'),'&#10;')">
            <xsl:for-each select="tokenize(.,'&quot;')">
                <xsl:value-of select="if (position() mod 2 = 0) 
                  then concat('&quot;',translate(.,' ','_'),'&quot;') else ."></xsl:value-of>
            </xsl:for-each>
            <xsl:text>&#10;</xsl:text>
        </xsl:for-each>
    </xsl:template>    
</xsl:stylesheet>

To test if, just get saxon.jar on sourceforge and use the following command line :

java -jar saxon9.jar -it:init regexp.xsl

The xslt file include the reference to the f.txt, the text file must be in the same directory as the xslt file. That can be easily changed by giving a parameter to the stylesheet.

It works in one pass.

Answer 3

This would be really easy if the quoted text was all on separate lines. So one approach is to split the text so you'll have that, do the easy transform, then rebuild the lines.

Splitting the text is easy, but we'll need to distinguish between newlines that were

1. already present in the file
2. added by us

To do that, we can end each line with a symbol indicating to which class it belongs. I'll just use 1 and 2, corresponding directly to the above. In sed, we have:

sed -e 's/$/1/' -e 's/"[^"]*"/2\n&2\n/g'

This produces:

2
"a aa"2
  MM  2
"bbb  b"2
1
MM    MM1
MM2
"b b "2
1

That's easy to transform, just use

sed -e '/".*"/ s/ /_/g' 

giving

2
"a_aa"2
  MM  2
"bbb__b"2
1
MM    MM1
MM2
"b_b_"2
1

Finally, we need to put it back together. This is actually pretty horrible in sed, but feasible using the hold space:

sed -e '/1$/ {s/1$//;H;s/.*//;x;s/\n//g}' -e '/2$/ {s/2$//;H;d}'

(This would be a lot clearer in, eg, awk.)

Pipe those three steps together and you're done.

Answer 4

These might work for you:

 sed 's/^/\n/;:a;s/\(\n[^"]*"[^ "]*\) \([^"]*"\)\n*/\1_\2\n/;ta;s/\n//;ta;s/\n//' file

Explanation:

Prepend a \\n to the start of the line, this will be used to bump along the substitutions. Replace a single with a _ within the " 's and whilst it's there place a \\n ready for the next round of substitutions. Having replaced all the 's, delete the \\n and repeat. When all substitutions have occured delete the \\n delimiter.

or this:

sed -r ':a;s/"/\n/;s/"/\n/;:b;s/(\n[^\n ]*) ([^\n]*\n)/\1_\2/g;tb;s/\n/%%%/g;ta;s/%%%/"/g' file

Explanation:

Replace the first set of "" 's with \\n 's. Replace first space between newlines with _ , repeat. Replace \\n 's with a unique delimiter ( %%% ), repeat from the beginning. Tidy up at the end by replacing all %%% with " 's.

A third way:

sed 's/"[^"]*"/\n&\n/g;$!s/$/@@@/' file |
sed '/"/y/ /_/;1{h;d};H;${x;s/\n//g;s/@@@/\n/g;p};d'

Explanation:

Surround all quoted expressions ( "..." ) with newlines ( \\n 's). Insert an end-of-line delimiter @@@ on all but the last line. Pipe result to second sed command. Translate all 's to _ 's for lines with a " in them. Store every line in the hold space (HS). At end of file, swap to the HS and delete all \\n 's and replace end-of-line delimiters with \\n 's

lastly:

sed 's/\("[^"]*"\)/$(tr '"' ' '_'"'<<<'"'"'\1'"'"')/g;s/^/echo /' file | sh

or GNU sed:

sed 's/\("[^"]*"\)/$(tr '"' ' '_'"'<<<'"'"'\1'"'"')/g;s/^/echo /e' file

left for the reader to work out.