Regex not matching data and dates

Question

I have an SQL Select dump with many lines each looks like this:

07/11/2011 16:48:08,07/11/2011 16:48:08,'YD','MANUAL',0,1,'text','text','text','text',,,,'text',,,0,0,

I want to do 2 things to each line:

1. Replace all dates with Oracle's sysdate function. Dates can also come without hour (like 07/11/2011 ).
2. Replace all null values with null string

Here's my attempt:

$_ =~ s/,(,|\n)/,null$1/g;                  # Replace no data by "null"
$_ =~ s/\d{2}\/\d{2}\/d{4}.*?,/sysdate,/g;  # Replace dates by "sysdate"

But this would transform the string to:

07/11/2011 16:48:08,07/11/2011 16:48:08,'YD','MANUAL',0,1,'text','text','text','text',null,,null,'text',null,,0,0,null

while I expect it to be

sysdate,sysdate,'YD','MANUAL',0,1,'text','text','text','text',null,null,null,'text',null,null,0,0,null

I don't understand why dates do not match and why some ,, are not replaced by null .

Any insights welcome, thanks in advance.

Answer 1

You could do this:

$ cat perlregex.pl
use warnings;
use strict;

my $row = "07/11/2011 16:48:08,07/11/2011 16:48:08,'YD','MANUAL',0,1,'text','text','text','text',,,,'text',,,0,0,\n";

print( "$row\n" );
while ( $row =~ /,([,\n])/ ) { $row =~ s/,([,\n])/,null$1/; }
print( "$row\n" );
$row =~ s/\d{2}\/\d{2}\/\d{4}.*?,/sysdate,/g;
print( "$row\n" );

Which results in this:

$ ./perlregex.pl
07/11/2011 16:48:08,07/11/2011 16:48:08,'YD','MANUAL',0,1,'text','text','text','text',,,,'text',,,0,0,

07/11/2011 16:48:08,07/11/2011 16:48:08,'YD','MANUAL',0,1,'text','text','text','text',null,null,null,'text',null,null,0,0,null

sysdate,sysdate,'YD','MANUAL',0,1,'text','text','text','text',null,null,null,'text',null,null,0,0,null

This could certainly be optimized, but it gets the point across.

Answer 2

\\d{2}\\/\\d{2}\\/d{4}.*?, didn't work because the last d wasn't escaped.
If a , can be on either side, or begin/end of string, you could do it in 2 steps:

step 1
s/(?:^|(?<=,))(?=,|\\n)/null/g
expanded:

/
  (?:  ^           # Begining of line, ie: nothing behind us
     | (?<=,)      # Or, a comma behind us
  )
     # we are HERE!, this is the place between characters
  (?=  ,           # A comma in front of us
     | \n          # Or, a newline in front of us
  )
/null/g
# The above regex does not consume, it just inserts 'null', leaving the
# same search position (after the insertion, but before the comma).

# If you want to consume a comma, it would be done this way:
s/(?:^|(?<=,))(,|\n)/null$1/xg
# Now the search position is after the 'null,'

step 2
s/(?:^|(?<=,))\\d{2}\\/\\d{2}\\/\\d{4}.*?(?=,|\\n)/sysdate/g

Or, you could combine them into a single regex, using the eval modifier:
$row =~ s/(?:^|(?<=,))(\\d{2}\\/\\d{2}\\/\\d{4}.*?|)(?=,|\\n)/ length $1 ? 'sysdate' : 'null'/eg;

Broken down it looks like this

s{
   (?: ^ | (?<=,) )  # begin of line or comma behind us
   (                 # Capt group $1
       \d{2}/\d{2}/\d{4}.*?     # date format and optional non-newline chars
     |                          # Or, nothing at all
   )                 # End Capt group 1
  (?= , | \n )       # comma or newline in front of us
}{
   length $1 ? 'sysdate' : 'null'
}eg  

If there is a chance of non-newline whitespace padding, it could be written as:

$row =~ s/(?:^|(?<=,))(?:([^\\S\\n]*\\d{2}\\/\\d{2}\\/\\d{4}.*?)|[^\\S\\n]*)(?=,|\\n)/ defined $1 ? 'sysdate' : 'null'/eg;

Answer 3

You want to replace something. Usually lookaheads are a better option for this :

$subject =~ s/(?<=,)(?=,|$)/null/g;

Explanation :

"
(?<=       # Assert that the regex below can be matched, with the match ending at this position (positive lookbehind)
   ,          # Match the character “,” literally
)
(?=        # Assert that the regex below can be matched, starting at this position (positive lookahead)
              # Match either the regular expression below (attempting the next alternative only if this one fails)
      ,          # Match the character “,” literally
   |          # Or match regular expression number 2 below (the entire group fails if this one fails to match)
      \$          # Assert position at the end of the string (or before the line break at the end of the string, if any)
)
"

Secodnly you wish to replace the dates :

$subject =~ s!\d{2}/\d{2}/\d{4}.*?(?=,)!sysdate!g;

That's almost the same with your original regex. Just replace the last , with lookahead. (If you don't want to replace it , don't match it.)

# \d{2}/\d{2}/\d{4}.*?(?=,)
# 
# Match a single digit 0..9 «\d{2}»
#    Exactly 2 times «{2}»
# Match the character “/” literally «/»
# Match a single digit 0..9 «\d{2}»
#    Exactly 2 times «{2}»
# Match the character “/” literally «/»
# Match a single digit 0..9 «\d{4}»
#    Exactly 4 times «{4}»
# Match any single character that is not a line break character «.*?»
#    Between zero and unlimited times, as few times as possible, expanding as needed (lazy) «*?»
# Assert that the regex below can be matched, starting at this position (positive lookahead) «(?=,)»
#    Match the character “,” literally «,»

Answer 4

Maybe .*? is too greedy, try:

$_ =~ s/\d{2}\/\d{2}\/d{4}[^,]+,/sysdate,/g;