使用SLR / LR / LALR解析器解析报价

Question

I'm trying to parse a list that is separated by spaces, but may include quotes that I have to treat as literals. So I tried to write a grammar and parse it using my favorite parsing algorithm, but I can't seem to get the grammar right.

The particular thing that makes it tricky is that I have to handle the " "" " case, which should be interpreted as one string with two quotes, whereas "" "" should be two empty strings.

To make the problem worse, I have to handle single quotes ' ' and comments bracketed by * * . Things like: ' * ' " * ' " are allowed, which should parse to * and * ' .

Is this just outright impossible or is there a grammar to do it?

The best try I've managed to come up with ( _ denotes space):

start -> argv $
argv -> argv _ term | term
term -> "" | '' | ** | "dqexpr" | 'sqexpr' | *comment* | expr
expr -> string without ", ', *, or _
dqexpr -> string without "
sqexpr -> string without '
comment -> string without *

But I can't be make this work on " "" " with a LR(1)/SLR(1) parser.

The grammar I attempted, for the simple case with no comments and only one quote type:

START -> ARGV $
ARGV -> ARGV _ TERM
ARGV -> TERM
TERM -> q STRING q
TERM -> FREE
STRING -> STRING CHAR
CHAR -> ''
CHAR -> q
CHAR -> c
FREE -> FREE c
FREE -> c

Here, '' is epsilon; q represents a quote, _ a space, and c any other character. The grammar can be attempted using the on-line tool at http://jsmachines.sourceforge.net/machines/slr.html

Answer 1

The STRING non-terminal in your attempted grammar is useless (that is, it cannot derive any string of terminals) because it has no non-recursive production. So a parser generator should discard it, along with the TERM -> q STRING q production. (Ideally, an error message to that effect would be generated.) If that were fixed, the production CHAR -> '' would generate an ambiguity, because a STRING can be any number of CHAR s, and you cannot tell how many epsilons there are in an empty string. Ideally, the parser generator would provide meaningful error messages although, as can be seen, not all do so.

That can be fixed by simply changing CHAR -> '' to STRING -> '' , which will also solve the conflict generated by the ambiguity of concatenating two epsilons. What remains is allowing a quoted STRING to contain a q , which contradicts the description in the pseudocode (" dqexpr -> string without " ").

If the intention is to allow a term to be a concatenation of quoted strings, so that " "" " is valid (without entering into its semantics), that can be done by adding another iteration non-terminal:

QTERMS -> QTERM
QTERMS -> QTERMS QTERM
QTERM  -> q STRING q

and changing TERM -> q STRING q to TERM -> QTERM .

My suspicion is that the desire is a simplified form of shell word processing, in which a "word" can be a concatenation of any number of terms, so that not only is " "" " a legal word, but so are " "' ' and "x"foo'y' . This allows the possibility to include both single and double quotes in the same word: "'"'"' . If we also make the assumption that a comment is equivalent to white-space, we end up with the following grammar:

START -> ARGV $
ARGV -> WORD
ARGV -> ARGV WHITES WORD
WHITES -> WHITE
WHITES -> WHITES WHITE
WHITE -> _
WHITE -> star CSTRING star
WORD -> TERM
WORD -> WORD TERM
TERM -> c
TERM -> squote SQSTRING squote
TERM -> dquote DQSTRING dquote
SQSTRING -> ''
SQSTRING -> SQSTRING c
SQSTRING -> SQSTRING _
SQSTRING -> SQSTRING star
SQSTRING -> SQSTRING dquote
DQSTRING -> ''
DQSTRING -> DQSTRING c
DQSTRING -> DQSTRING _
DQSTRING -> DQSTRING star
DQSTRING -> DQSTRING squote
CSTRING  -> ''
CSTRING  -> CSTRING c
CSTRING  -> CSTRING _
CSTRING  -> CSTRING squote
CSTRING  -> CSTRING squote

The SLR tool you use successfully generates a parser using the above grammar.

Answer 2

The grammar should actually be the same as matching parentheses, so you should be able to find lots of examples on parsing math expressions.

I'm not exactly sure what your example grammar is trying to accomplish but including "", '', and ** in your term rule seems odd to me. Try changing it to be something like:

expr -> string
expr -> ' string '
expr -> " string "