ANTLR4 calling next step in visitor not working

Question

I have my grammar complete and want to implement now functionality. But I get it not working, I do not understand how to evaluate in the functionExpr which command should be called. As a test expression I am using

  isSet(value, test) and isSet(value, foo)

Currently I do not know how to evaluate the single CommandIsSet Function and also evaluate there the variable type. To solve this the order should be

1. visitProg
2. visitAndExpr
3. visitFunctionExpr (for first isSet(value, test)
4. visitCommandIsSet
5. visitVariableParamter
6. visitValueType
7. visitFunctionExpr (for second isSet(value, foo)
8. visitCommandIsSet
9. visitVariableParamter
10. visitValueType

Currently I stuck in step 3, how can I determine that its a commandIsSet do I need to use the ctx.getText() and check if it starts with isSet?

I have already found a great answer from @BartKiers in SO to such a topic Using Visitors in AntLR4 in a Simple Integer List Grammar from there I used the Value Object.

Here is my grammar

grammar FEL;

prog: expr+ EOF;
expr:
             statement                          #StatementExpr
             |NOT expr                          #NotExpr
             | expr AND expr                 #AndExpr
             | expr (OR | XOR) expr        #OrExpr
             | function                           #FunctionExpr
             | LPAREN expr RPAREN       #ParenExpr
             | writeCommand                 #WriteExpr
 ;

writeCommand: setCommand | setIfCommand;
statement: ID '=' getCommand  NEWLINE         #Assign;
setCommand: 'set' LPAREN variableType '|' parameter RPAREN SEMI;
setIfCommand: 'setIf' LPAREN variableType '|' expr '?' parameter ':' parameter RPAREN SEMI;

getCommand:         getFieldValue                   #FieldValue
                            | getInstanceAttribValue     #InstanceAttribValue
                            | getFormAttribValue          #FormAttributeValue
                            | getMandatorAttribValue   #MandatorAttributeValue
                            ;

getFieldValue: 'getFieldValue' LPAREN instanceID=ID COMMA fieldname=ID RPAREN;
getInstanceAttribValue: 'getInstanceAttrib' LPAREN instanceId=ID COMMA moduleId=ID COMMA attribname=ID RPAREN;
getFormAttribValue: 'getFormAttrib' LPAREN formId=ID COMMA moduleId=ID COMMA attribname=ID RPAREN;
getMandatorAttribValue: 'getMandatorAttrib' LPAREN mandator=ID COMMA moduleId=ID COMMA attribname=ID RPAREN;

twoParameterList: parameter '|' parameter;
parameter:
            variableType #VariableParameter
            | constType  #ConstParameter
            ;
pdixFuncton:ID;
constType:
                ID #ID_Without
                | '"'  ID '"' #ID_WITH
                ;
variableType:
                    valueType                      #ValueTyp
                    |instanceType                #InstanceTyp
                    |formType                      #FormTyp
                    |bufferType                    #BufferTyp
                    |instanceAttribType       #InstanceAttribTyp
                    |formAttribType             #FormAttribTyp
                    |mandatorAttribType     #ManatorTyp
                    ;
valueType:'value' COMMA parameter (COMMA functionParameter)?;
instanceType: 'instance' COMMA instanceParameter;
formType: 'form' COMMA formParameter;
bufferType: 'buffer' COMMA parameter;
instanceParameter: 'instanceId'
                                | 'instanceKey'
                                | 'firstpenId'
                                | 'lastpenId'
                                | 'lastUpdate'
                                | 'started'
                                ;
formParameter: 'formId'
                            |'formKey'
                            |'lastUpdate'
                            ;
functionParameter: 'lastPen'
                                | 'fieldGroup'
                                | ' fieldType'
                                | 'fieldSource'
                                | 'updateId'
                                | 'sessionId'
                                | 'icrConfidence'
                                | 'icrRecognition'
                                |  'lastUpdate';

instanceAttribType:('instattrib' | 'instanceattrib') COMMA attributeType;
formAttribType:'formattrib' COMMA attributeType;
mandatorAttribType: 'mandatorattrib' COMMA attributeType;
attributeType:ID '#' ID;

 function:
                commandIsSet                                    #IsSet
                | commandIsEmpty                            #IsEmpty
                | commandIsEqual                               #IsEqual
                |commandIsNumLessEqual                  #IsLessEqual
                |commandIsNumLess                           #IsNumLess
                |commandIsNumGreaterEqual            #IsNumGreaterEqual
                |commandIsNumGreater                    #IsNumGreater
                |commandIsNumEqual                       #IsNumEqual
                |commandIsLess                               #IsLess
                |commandIsLessEqual                       #IsLessEqual
                |commandIsGreater                           #IsGreater
                |commandIsGreaterEqual                  #IsGreaterEqual
                |commandMatches                             #Matches
                |commandContains                            #Contains
                |commandEndsWith                            #EndsWith
                |commandStartsWith                          #StartsWith
                ;
commandIsSet: IS_SET LPAREN parameter RPAREN;
commandIsEmpty: IS_EMPTY LPAREN parameter RPAREN;
commandIsEqual: IS_EQUAL LPAREN twoParameterList RPAREN;
commandStartsWith: 'startsWith' LPAREN twoParameterList RPAREN;
commandEndsWith: 'endsWith' LPAREN twoParameterList RPAREN;
commandContains: 'contains' LPAREN twoParameterList RPAREN;
commandMatches: 'maches' LPAREN twoParameterList RPAREN;
commandIsLess: 'isLess' LPAREN twoParameterList RPAREN;
commandIsLessEqual: 'isLessEqual' LPAREN twoParameterList RPAREN;
commandIsGreater: 'isGreater' LPAREN twoParameterList RPAREN;
commandIsGreaterEqual: 'isGreaterEqual' LPAREN twoParameterList RPAREN;
commandIsNumEqual: 'isNumEqual' LPAREN twoParameterList RPAREN;
commandIsNumGreater: 'isNumGreater' LPAREN twoParameterList RPAREN;
commandIsNumGreaterEqual: 'isNumGreaterEqual' LPAREN twoParameterList RPAREN;
commandIsNumLess: 'isNumLess' LPAREN twoParameterList RPAREN;
commandIsNumLessEqual: 'isNumLessEqual' LPAREN twoParameterList RPAREN;

/*
stringFunctionType:
    a=substringStrFunction
    |   a=cutStrFunction
    |   a=replaceStrFunction
    |   a=reformatDateStrFunction
    |   a=translateStrFunction
    |   a=fillStrFunction
    |   a=concatStrFunction
    |   a=justifyStrFunction
    |   a=ifElseStrFunction
    |   a=tokenStrFunction
    |   a=toLowerFunction
    |   a=toUpperFunction
    |   a=trimFunction
;
*/

LPAREN : '(';
RPAREN : ')';
LBRACE : '{';
RBRACE : '}';
LBRACK : '[';
RBRACK : ']';
SEMI : ';';
COMMA : ',';
DOT : '.';
ASSIGN : '=';
GT : '>';
LT : '<';
BANG : '!';
TILDE : '~';
QUESTION : '?';
COLON : ':';
EQUAL : '==';
LE : '<=';
GE : '>=';
NOTEQUAL : '!=';
AND : 'and';
OR : 'or';
XOR :'xor';
NOT :'not'  ;
INC : '++';
DEC : '--';
ADD : '+';
SUB : '-';
MUL : '*';
DIV : '/';

INT: [0-9]+;
NEWLINE: '\r'? '\n';
IS_SET:'isSet';
IS_EMPTY:'isEmpty';
IS_EQUAL:'isEqual';
WS: (' '|'\t' | '\n' | '\r' )+ -> skip;

ID
:   JavaLetter JavaLetterOrDigit*
;

fragment
JavaLetter
:   [a-zA-Z$_] // these are the "java letters" below 0xFF
|   // covers all characters above 0xFF which are not a surrogate
    ~[\u0000-\u00FF\uD800-\uDBFF]
    {Character.isJavaIdentifierStart(_input.LA(-1))}?
|   // covers UTF-16 surrogate pairs encodings for U+10000 to U+10FFFF
    [\uD800-\uDBFF] [\uDC00-\uDFFF]
    {Character.isJavaIdentifierStart(Character.toCodePoint((char)_input.LA(-2), (char)_input.LA(-1)))}?
;

fragment
JavaLetterOrDigit
:   [a-zA-Z0-9$_] // these are the "java letters or digits" below 0xFF
|   // covers all characters above 0xFF which are not a surrogate
    ~[\u0000-\u00FF\uD800-\uDBFF]
    {Character.isJavaIdentifierPart(_input.LA(-1))}?
|   // covers UTF-16 surrogate pairs encodings for U+10000 to U+10FFFF
    [\uD800-\uDBFF] [\uDC00-\uDFFF]
    {Character.isJavaIdentifierPart(Character.toCodePoint((char)_input.LA(-2), (char)_input.LA(-1)))}?
;
fragment DoubleQuote: '"' ;   // Hard to read otherwise.

This is my visitor implementation with a test expression

public class FELCustomVisitor extends FELBaseVisitor<Value> {

// used to compare floating point numbers
public static final double SMALL_VALUE = 0.00000000001;

// store variables (there's only one global scope!)
private Map<String, Value> memory = new HashMap<String, Value>();

public static void main(String[] args) {
    String expression = "isSet(value, test) and isSet(value, foo)";
    FELLexer lexer = new FELLexer(new ANTLRInputStream(expression));
    FELParser parser = new FELParser(new CommonTokenStream(lexer));
    ParseTree tree = parser.prog();
    Value answer = new FELCustomVisitor().visit(tree);
    System.out.printf("answer is %s", answer.asBoolean());
}


@Override
public Value visitProg(FELParser.ProgContext ctx) {
    System.out.println("called prog " + ctx.getText());
    return this.visitChildren(ctx);
}

@Override
public Value visitAndExpr(FELParser.AndExprContext ctx) {
    Value left = this.visit(ctx.expr(0));
    Value right = this.visit(ctx.expr(2));
    System.out.println("working on and expression: left == " + left.asBoolean() + " right is " + right.asBoolean());
    return new Value(left.asBoolean() && right.asBoolean());
}

@Override
public Value visitFunctionExpr(FELParser.FunctionExprContext ctx) {
    System.out.println("called function expresssion " + ctx.getText());
    FELParser.FunctionContext function = ctx.function();

    return null;
}


@Override
public Value visitCommandIsSet(FELParser.CommandIsSetContext ctx) {
    //here I need to extract the inner variable type with this i can evaluate the return value

    return super.visitCommandIsSet(ctx);
}

@Override
public Value visitVariableParameter(FELParser.VariableParameterContext ctx) {
    return super.visitVariableParameter(ctx);
}
@Override
public Value visitValueTyp(FELParser.ValueTypContext ctx) {
    return super.visitValueTyp(ctx);
}

Answer 1

The part this.visit(ctx.expr(2)) in this method is wrong:

@Override
public Value visitAndExpr(FELParser.AndExprContext ctx) {
    Value left = this.visit(ctx.expr(0));
    Value right = this.visit(ctx.expr(2));
    System.out.println("working on and expression: left == " + left.asBoolean() + " right is " + right.asBoolean());
    return new Value(left.asBoolean() && right.asBoolean());
}

There are 2 child nodes that have index 0 and 1.

You're making it a bit too complicated in your visitor. Start with the most simple expressions you'd like to evaluate, like isSet(value, test) , and work your way from the bottom of the tree to the root of it and implement these visit...() methods. In case of isSet(value, test) that would be the rules:

@Override
public Value visitID_Without(FELParser.ID_WithoutContext ctx) {
    // ID #ID_Without
}

@Override
public Value visitCommandIsSet(FELParser.CommandIsSetContext ctx) {
    // commandIsSet: IS_SET LPAREN parameter RPAREN;
}

@Override
public Value visitProg(FELParser.ProgContext ctx) {
    // expr+ EOF
}

when that works, add the and expression to it to support isSet(value, test) and isSet(value, foo) :

@Override
public Value visitID_Without(FELParser.ID_WithoutContext ctx) {
    // ID #ID_Without
}

@Override
public Value visitCommandIsSet(FELParser.CommandIsSetContext ctx) {
    // commandIsSet: IS_SET LPAREN parameter RPAREN;
}

@Override
public Value visitAndExpr(FELParser.AndExprContext ctx) {
    // expr AND expr
}

@Override
public Value visitProg(FELParser.ProgContext ctx) {
    // expr+ EOF
}

etc.

A small working example would look like this:

public class FELCustomVisitor extends FELBaseVisitor<Value> {

    private final Map<String, Value> memory;

    public FELCustomVisitor(Map<String, Value> memory) {
        this.memory = memory;
    }

    @Override
    public Value visitID_Without(FELParser.ID_WithoutContext ctx) {
        // ID #ID_Without
        return this.memory.getOrDefault(ctx.ID().getText(), Value.NULL);
    }

    @Override
    public Value visitCommandIsSet(FELParser.CommandIsSetContext ctx) {
        // commandIsSet: IS_SET LPAREN parameter RPAREN;
        Value param = this.visit(ctx.parameter());
        return param == Value.NULL ? Value.FALSE : Value.TRUE;
    }

    @Override
    public Value visitAndExpr(FELParser.AndExprContext ctx) {
        // expr AND expr
        Value lhs = this.visit(ctx.expr(0));
        Value rhs = this.visit(ctx.expr(1));
        return lhs.asBoolean() && rhs.asBoolean() ? Value.TRUE : Value.FALSE;
    }

    @Override
    public Value visitProg(FELParser.ProgContext ctx) {
        // expr+ EOF
        Value result = Value.NULL;
        for (FELParser.ExprContext expr : ctx.expr()) {
            result = this.visit(expr);
        }
        return result;
    }

    public static void main(String[] args) {
        String[] expressions = {
                "isSet(value, test)",
                "isSet(value, foo)",
                "isSet(value, fooooooo)",
                "isSet(value, test) and isSet(value, foo)",
                "isSet(value, nope) and isSet(value, test) and isSet(value, foo)"
        };

        for (String expression  : expressions) {
            FELLexer lexer = new FELLexer(CharStreams.fromString(expression));
            FELParser parser = new FELParser(new CommonTokenStream(lexer));
            ParseTree tree = parser.prog();
            Map<String, Value> memory = new HashMap<>();
            memory.put("test", new Value(42));
            memory.put("foo", new Value("bar"));
            Value answer = new FELCustomVisitor(memory).visit(tree);
            System.out.printf("%s = %s%n", expression, answer);
        }
    }
}

class Value {

    public static final Value TRUE = new Value(true);
    public static final Value FALSE = new Value(false);
    public static final Value NULL = new Value(null);

    private final Object value;

    public Value(Object value) {
        this.value = value;
    }

    public Boolean asBoolean() {
        return (Boolean) this.value;
    }

    @Override
    public String toString() {
        return String.format("Value{type: %s, value: %s}",
                this.value == null ? "null" : this.value.getClass().getSimpleName(), this.value);
    }
}

when running it, you'll see the following printed to your console:

isSet(value, test) = Value{type: Boolean, value: true}
isSet(value, foo) = Value{type: Boolean, value: true}
isSet(value, fooooooo) = Value{type: Boolean, value: false}
isSet(value, test) and isSet(value, foo) = Value{type: Boolean, value: true}
isSet(value, nope) and isSet(value, test) and isSet(value, foo) = Value{type: Boolean, value: false}

EDIT

[...] So visitVariableParameter will be called, how can I distinguish there between the different options and call the correct visit method [...]

You don't need to override that method. Let's say you want to implement isSet(instance, instanceId) , then all you have to do is override visitInstanceType(...) in your visitor. Whenever somewhere in your code this.visit(ctx.parameter()); is called, it will automatically makes sure the children of parameter are called.

You only add this method to the visitor:

@Override
public Value visitInstanceType(FELParser.InstanceTypeContext ctx) {
    // 'instance' COMMA instanceParameter
    Value value = new Value(String.format("TODO: instance, `%s`", ctx.instanceParameter().getText()));
    System.out.println("We're in visitInstanceType(...), returning: " + value);
    return value;
}

And now when evaluating "isSet(instance, instanceId)" , you will see that the above method is invoked properly.

Answer 2

Thanks for the great answer, this helps a lot. Actually I am testing this, but how to handle the rule

variableType:
                    valueType                      #ValueTyp
                    |instanceType                #InstanceTyp
                    |formType                      #FormTyp
                    |bufferType                    #BufferTyp
                    |instanceAttribType       #InstanceAttribTyp
                    |formAttribType             #FormAttribTyp
                    |mandatorAttribType     #ManatorTyp
                    ;

This will called in commandIsSet because there is a parameter. So visitVariableParameter will be called, how can I distinguish there between the different options and call the correct visit method. Actually I am trying this with ctx.start.toString().equals("value") but this seem not a good approach

@Override
public Value visitVariableParameter(FELParser.VariableParameterContext ctx) {
    System.out.println("called variable parameter");
    
    if (ctx.start.toString().startsWith("value")){
        //compile error wrong ctx
        return this.visitValueTyp(ctx);
    }
    
    return null;
}