Python PLY：在每一行输入中获取语法错误

Question

Working on writing a compiler for the for loop construct of C. However, Im still stuck at the preliminary task of parsing the starting part of a C program namely the header files to be included and the main function.

Here is my code:

import ply.lex as lex
import ply.yacc as yacc
tokens = ('HASH','INCLUDE','HEADER_FILE','MAIN','FLOW_OPEN','FLOW_CLOSE','SEMI_COLON','TYPE','SMALL_OPEN','SMALL_CLOSE','OTHERS')

t_HASH = r'\#'
t_INCLUDE = r'include'
t_HEADER_FILE = r'<stdio.h>'
t_MAIN = r'main' 
t_FLOW_OPEN = r'{'
t_FLOW_CLOSE = r'}'
t_SMALL_OPEN = r'\('
t_SMALL_CLOSE = r'\)'
t_SEMI_COLON = r';'
t_OTHERS = r'[a-zA-Z][a-zA-Z]*'
t_TYPE = r'int|void'

def t_error(token):
    print(f'Illegal character: {token.value}')

def t_whitespace(t):
    r'\s+'
    pass

def t_newline(t):
    r'\n+'
    t.lexer.lineno += len(t.value)

lexer = lex.lex()
#Building the parser

def p_expression_start(p):
    'expression : header body'

def p_header(p):
    'header : HASH INCLUDE HEADER_FILE'

def p_body(p):
    'body : main rest'

def p_main(p):
    'main : TYPE MAIN SMALL_OPEN SMALL_CLOSE'

def p_rest(p):
    'rest : FLOW_OPEN st FLOW_CLOSE'

def p_st(p):
    ''''
        st : OTHERS st
            | end
        '''
def p_end(p): #Empty production
    'end : SEMI_COLON' 

def p_error(p):
    print("Syntax error in input!")

parser = yacc.yacc(method='LALR',debug=True)

with open(r'forparsing.txt','r') as file:
    while True:
        try:
            line = next(file)
            print('Parsing')
            parser.parse(line)
        except:
            print('Finished')
            break

And the input I'm giving is:

# include <stdio.h>
void main()
{
 abc;
 }

But on running the program, I get a Syntax error on each line. What could be wrong here. From my understanding, the parser is not able to derive back the start symbol from the given input but I dont know how to go about fixing this. In general, how do I debug syntax error issues with PLY?

Answer 1

None of your lines of input are syntactically valid on their own. They only form a syntactically valid program when parsed as a whole. So you'll need to call parse once with a string containing the whole program, not once per line.

You can do this by just calling file.read() in your file handling code instead of using a while loop.

---

The syntax error you're running into after fixing this is due to the way the that overlapping lexical rules are handled in PLY. In sane lexer generators, the rule that produces the longest match wins and, if both produce the same match, the one that comes first in the code wins. However, in PLY the one with the longest regex wins. Due to this behavior, you can't use separate rules to match identifiers and keywords using PLY. In this case, the t_OTHERS rule is used even if, say, t_INCLUDE also matches.

Instead the PLY documentation recommends the following way of matching identifiers and keywords:

To handle reserved words, you should write a single rule to match an identifier and do a special name lookup in a function like this:

  reserved = { 'if' : 'IF', 'then' : 'THEN', 'else' : 'ELSE', 'while' : 'WHILE', ... } tokens = ['LPAREN','RPAREN',...,'ID'] + list(reserved.values()) def t_ID(t): r'[a-zA-Z_][a-zA-Z_0-9]*' t.type = reserved.get(t.value,'ID') # Check for reserved words return t 

This approach greatly reduces the number of regular expression rules and is likely to make things a little faster.

Note: You should avoid writing individual rules for reserved words. For example, if you write rules like this,

  t_FOR = r'for' t_PRINT = r'print' 

those rules will be triggered for identifiers that include those words as a prefix such as "forget" or "printed". This is probably not what you want.

Again, it should be pointed out, that neither of the issues mentioned there exist in lexer generators that use the maximum munch rule.

---

In general, how do I debug syntax error issues with PLY?

The first step would be to change p_error to print out some useful information (such as which type of token on which line caused the syntax error) like this:

def p_error(p):
    if p == None:
        token = "end of file"
    else:
        token = f"{p.type}({p.value}) on line {p.lineno}"

    print(f"Syntax error: Unexpected {token}")