Java - How to handle parse tree recursion for leaf nodes?

Question

First of all, sorry for the bad title. I don't really know what to call this problem.

I am trying to code a parser for my Turtle graphics program (if you don't know what it is, you basically create a small programming language made up of commands that are used to direct the movements of a "turtle". For example, the input "FORW 5. LEFT 45." would make the turtle move 5 steps forward and then rotate 45 degrees to the left.)

My BNF grammar looks like this:

<EXPR>::= <CMD><EXPR> | <EOF>

<CMD>::=
  <FORW><INT><PERIOD>
| <BACK><INT><PERIOD>
| <LEFT><INT><PERIOD>
| <RIGHT><INT><PERIOD>
| <DOWN><PERIOD>
| <UP><PERIOD>
| <COLOR><HEX><PERIOD>
| <REP><INT><CMD>
| <REP><INT><QUOTE><EXPR><QUOTE>

(All <-.-> are terminals [tokens] except for <EXPR> and <CMD> .)

The REP command repeats a set of commands (the commands within the quote symbols) X amount of times. For example: REP 3 "FORW 5. LEFT 45." repeats the commands FORW 5. LEFT 45. 3 times.

This is the command (REP) that I am having problems with implementing into my parser (since it contains non-terminals in its grammar).

Parse Tree:

// Ett syntaxträd
abstract class ParseTree {
    abstract public int evaluate();
}

// Ett syntaxträd som representerar ett tal
class ExprNode extends ParseTree {
    ParseTree left, right;
    public ExprNode(ParseTree left, ParseTree right) {
        this.left = left;
        this.right = right;
    }
    public int evaluate() {
        return 0;
    }
}

// Ett syntaxträd som representerar någon av de fyra binära operationerna
class CMDNode extends ParseTree {
    TokenType cmd;
    int num;
    String hex;
    ParseTree child;

    public CMDNode(TokenType cmd) {
        this.cmd = cmd;
    }

    public CMDNode(TokenType cmd, int num) {
        this.cmd = cmd;
        this.num = num;
    }

    public CMDNode(TokenType cmd, String hex) {
        this.cmd = cmd;
        this.hex = hex;
    }

    public CMDNode(TokenType cmd, ParseTree child) {
        this.cmd = cmd;
        this.child = child;
    }

    public int evaluate() {
        return 0;
    }
}

(Sorry for the comments being in Swedish)

Parser:

/**
 * En rekursiv medåknings-parser för aritmetiska uttryck.
 * Se README för mer info.
 *
 * 
 */
public class Parser {
    private Lexer lexer;

    /** Variabler för att kunna ge pratig förklaring av vad som händer
     * i parsningen.  Om man inte har behov av denna feature kan koden
     * som relaterar till dessa variabler tas bort.
     */
    private boolean verbose;
    private int depth;

    /** Om verbose är satt till sann kommer Parsern att prata en massa
     * medans den gör sitt jobb.
     */
    public Parser(Lexer lexer, boolean verbose) {
        this.lexer = lexer;
        this.verbose = verbose;
    }

    private void talk(String s) {
        if (verbose)
            System.out.printf("%"+(3*depth+1)+"s%s\n", "", s);
    }

    public ParseTree parse() throws SyntaxError {
        // Startsymbol är Expr
        depth = 0;
        talk("Start parse()");
        ++depth;
        ParseTree result = Expr();
        // Borde inte finnas något kvar av indata när vi parsat ett uttryck
        if (lexer.nextToken().getType() != TokenType.EOF) {
            throw new SyntaxError();
        }
        return result;
    }

    private ParseTree Expr() throws SyntaxError {
        //talk("Enter Expr()");
        //++depth;
        ParseTree result = Cmd();
        //talk("[Expr()] Read cmd done");
        while (lexer.peekToken().getType() == TokenType.FORW ||
               lexer.peekToken().getType() == TokenType.BACK||
               lexer.peekToken().getType() == TokenType.LEFT||
               lexer.peekToken().getType() == TokenType.RIGHT||
               lexer.peekToken().getType() == TokenType.UP||
               lexer.peekToken().getType() == TokenType.DOWN||
               lexer.peekToken().getType() == TokenType.COLOR||
               lexer.peekToken().getType() == TokenType.REP) {
            ParseTree expression = Expr();
            //talk("[Expr()] Read operator " + operator);

            //talk("[Expr()] Read term done");
            result = new ExprNode(result, expression);
        }
        //--depth;
        //talk("Leave Expr()");
        return result;
    }

    private ParseTree Cmd() throws SyntaxError  {
        Token t = lexer.nextToken();
        if(t.getType() == TokenType.FORW || t.getType() == TokenType.BACK || t.getType() == TokenType.LEFT || t.getType() == TokenType.RIGHT) {
            Token num = lexer.nextToken();
            if(num.getType() != TokenType.DECIMAL) {
                throw new SyntaxError();
            }
            if(lexer.nextToken().getType() != TokenType.PERIOD) {
                throw new SyntaxError();
            }
            return new CMDNode(t.getType(), (Integer)num.getData());
        }
        else if(t.getType() == TokenType.UP || t.getType() == TokenType.DOWN) {
            if(lexer.nextToken().getType() != TokenType.PERIOD) {
                throw new SyntaxError();
            }
            return new CMDNode(t.getType());
        }
        else if(t.getType() == TokenType.COLOR) {
            Token hex = lexer.nextToken();
            if(hex.getType() != TokenType.HEX) {
                throw new SyntaxError();
            }
            if(lexer.nextToken().getType() != TokenType.PERIOD) {
                throw new SyntaxError();
            }
            return new CMDNode(t.getType(), (String)hex.getData());
        }
        else if(t.getType() == TokenType.REP) {
            Token num = lexer.nextToken();
            if(num.getType() != TokenType.DECIMAL) {
                throw new SyntaxError();
            }
            if(lexer.peekToken().getType() == TokenType.QUOTE) {
                Expr();
            }
            else {
                Cmd();
            }
        }
        else {
            throw new SyntaxError();
        }
        return null;

    }

}

I added all of the code but the most important part (the part that I am struggling with) is at the bottom of the Parser code when I am trying to handle the REP token.

else if(t.getType() == TokenType.REP) {
            Token num = lexer.nextToken();
            if(num.getType() != TokenType.DECIMAL) {
                throw new SyntaxError();
            }
            if(lexer.peekToken().getType() == TokenType.QUOTE) {
                Expr();
            }
            else {
                Cmd();
            }

According to the grammar, REP can have two "outcomes". It can either repeat only one command X amount of times or a set of commands X amount of times (I identify the set with the help of the QUOTE tokens). But I have no idea what to do or what to write within the second if and else statements. I currently just added a recursive call to either the Expr() function or the Cmd() function but I think that this is wrong. I think that this will instead create a second Parse tree instead of a node that is connected to the current CMDNode. I have no idea how to solve this.

Sorry for the long and crappy explanation but I hope you have understood what the issue is.

Just as a sidenote, I have not had much earlier experience with Tree structures so this problem might seem silly to you :)

Thanks!

Answer 1

This is untested pseudo code but it should be enough to help you continue your development.

        if(lexer.peekToken().getType() == TokenType.QUOTE) {
            // Here we know there will be one or more children and that the sequence starts and ends with a "quote command"
            List<ParseTree> children = new ArrayList<>();
            ParseTree child = Cmd(); // The initial "quote command" - just ignore
            while ((child = Cmd()) != TokenType.QUOTE) {
                // Will stop looping when the second quote is found
                children.add(child);
            }
            return new CMDNode(t.getType(), children); // Yes, you need to create a new constructor
        }
        else {
            // Here we know there will only one child
            ParseTree child = Cmd();
            return new CMDNode(t.getType(), child);
        }

You also need to add a new "quote command":

// Existing code
else if(t.getType() == TokenType.REP) {
     ...
}
// New code starts here
else if(t.getType() == TokenType.QUOTE) {
    // This "command" is only used when parsing the input string.
    return new CMDNode(t.getType());
}
// Existing code again
else {
    throw new SyntaxError();
}