使用解析器重新同步处理PLY.yacc错误

Question

I'm trying implement a user-friendly syntax error handling during the parsing. From what I've observed in the official PLY documentation . One way is to raise an exception when the first SyntaxError occurs and terminate the parsing. However I would like to do something similar, as the documentation suggests, to use the parser resynchronization technique.

The documentation says:

The most well-behaved approach for handling syntax errors is to write grammar rules that include the error token. For example, suppose your language had a grammar rule for a print statement like this:

 def p_statement_print(p): 'statement : PRINT expr SEMI' ... 

To account for the possibility of a bad expression, you might write an additional grammar rule like this:

 def p_statement_print_error(p): 'statement : PRINT error SEMI' print("Syntax error in print statement. Bad expression") 

I have a grammar excerpt like this:

def p_operation(self, p) -> None:
    '''
    operation : unaryOperation
              | binaryOperation
    '''

def p_unaryOperation(self, p) -> None:
    '''
    unaryOperation : unaryOperation L_SQUARE_BRACKET projection R_SQUARE_BRACKET
                   | RELATION_NAME
    '''

def p_projection(self, p) -> None:
    '''
    projection : multipleAttributes
               | attribute
    '''

def p_multipleAttributes(self, p) -> None:
    '''
    multipleAttributes : projection COMMA attribute
    '''

def p_attribute(self, p) -> None:
    '''
    attribute : ATTRIBUTE
    '''

I'm quite unsure how should I define such new rules including the error token. Should I replace every non-terminal with the error token?

Looking forward to seeing your replies! Thanks a lot for your help

Answer 1

You definitely should not add an error production for every non-terminal.

Resynchronisation works when there is some token which would normally reset the parsing context to a known state. In languages with a clear end-of-statement marker -- a semicolon in the example you cite -- that token works well as a resynchronisation point. Discarding text up to the next semicolon and then parsing from there won't work 100% of the time, but it does work in many cases.

Parentheses and brackets can also be used as resynchronisation points, but the heuristic is not as reliable, because many syntax errors are the result of mismatched parentheses or brackets. Scanning for a missing close bracket could discard the entire input, for example.

Resynchronisation is more complicated in the case of languages without clear statement delimiters, including languages like Python where a newline only terminate statements if they are not nested within parentheses. Discarding up to a newline might work, but you might have to deal with feedback between the scanner and the parser which determines when a newline is transmitted as a token and when it is skipped as whitespace.

Inconsistent indentation can be a useful resynchronisation trigger, with a couple of caveats. First, you must not reject valid input with "misleading" indentation, so the trigger needs to be more sensitive during resynchronisation than during normal parsing. Second, tracking inconsistent indentation definitely requires a parser->scanner back-channel. So it's more work than simple panic recovery, but it can be effective.

The bottom line is that there are few, if any, universal algorithms for good error reporting and recovery. You need to base your strategy on the syntactic nature of the language.

Ideally, you will want to refine the code by examining your parser's response to common errors, but that can't really be done until you have an actual deployment and can see what the common errors are. So the best advice I can give is to start with a simple recovery strategy and see how it does with different syntax errors, particularly the syntax errors you accidentally created (or those of your friends and collaborators). Keep an archive of different syntax errors encountered, which you can use to test improvements to your diagnosis and recovery code. Don't expect it to be perfect, since it is a difficult problem, but do try to make it more accurate whenever you can.