解析单引号或双引号并允许使用正则表达式转义字符（在Python中）

Question

I have input that looks like a list of arguments:

input1 = '''
title="My First Blog" author='John Doe'
'''

The values can be surrounded by single or double quotes, however, escaping is also allowed:

input2 = '''
title='John\'s First Blog' author="John Doe"
'''

Is there a way to use regular expressions to extract the key value pairs accounting for either single or double quotes and escaped quotes?

Using python, I can use the following regular expression and handle the non-escaped quotes:

rex = r"(\w+)\=(?P<quote>['\"])(.*?)(?P=quote)"

The returns are then:

import re
re.findall(rex, input1)
[('title', '"', 'My First Blog'), ('author', "'", 'John Doe')]

and

import re
re.findall(rex, input2)
[('title', "'", 'John'), ('author', '"', 'John Doe')]

The latter being incorrect. I can't figure out how to handle escaped quotes--assumedly in the (.*?) section. I've been working with the solution in the posted answers on Python regex to match text in single quotes, ignoring escaped quotes (and tabs/newlines) to no avail.

Technically, I don't need findall to return the quote character--rather just the key/value pairs--but that is easily dealt with.

Any help would be appreciated! Thanks!

Answer 1

I think Tim's use of backreferences overcomplicates the expression and (guessing here) also makes it slower. The standard approach (used in the owl book) is to match single- and double-quoted strings separately:

rx = r'''(?x)
    (\w+) = (
        ' (?: \\. | [^'] )* '
        |
        " (?: \\. | [^"] )* "
        |
        [^'"\s]+
    )
'''

Add a bit of postprocessing and you're fine:

input2 = r'''
title='John\'s First Blog' author="John Doe"
'''

data = {k:v.strip("\"\'").decode('string-escape') for k, v in re.findall(rx, input2)}
print data
# {'author': 'John Doe', 'title': "John's First Blog"}

As a bonus, this also matches unquoted attributes like weight=150 .

Add: here's a cleaner way without regular expressions:

input2 = r'''
title='John\'s First Blog' author="John Doe"
'''

import shlex

lex = shlex.shlex(input2, posix=True)
lex.escapedquotes = '\"\''
lex.whitespace = ' \n\t='
for token in lex:
    print token

# title
# John's First Blog
# author
# John Doe

Answer 2

EDIT

My inital regex solution had a bug in it. That bug masked an error in your input string: input2 is not what you think it is:

>>> input2 = '''
... title='John\'s First Blog' author="John Doe"
... '''
>>> input2      # See - the apostrophe is not correctly escaped!
'\ntitle=\'John\'s First Blog\' author="John Doe"\n'  

You need to make input2 a raw string (or use double backslashes):

>>> input2 = r'''
... title='John\'s First Blog' author="John Doe"
... '''
>>> input2
'\ntitle=\'John\\\'s First Blog\' author="John Doe"\n'

Now you can use a regex that handles escaped quotes correctly:

>>> rex = re.compile(
    r"""(\w+)# Match an identifier (group 1)
    =        # Match =
    (['"])   # Match an opening quote (group 2)
    (        # Match and capture into group 3:
     (?:     # the following regex:
      \\.    # Either an escaped character
     |       # or
      (?!\2) # (as long as we're not right at the matching quote)
      .      # any other character.
     )*      # Repeat as needed
    )        # End of capturing group
    \2       # Match the corresponding closing quote.""", 
    re.DOTALL | re.VERBOSE)
>>> rex.findall(input2)
[('title', "'", "John\\'s First Blog"), ('author', '"', 'John Doe')]