ANTLR4 and the Python target

Question

I'm having issues getting going with a Python target in ANTLR4. There seems to be very few examples available and going to the corresponding Java code doesn't seem relevant.

I'm using the standard Hello.g4 grammar:

// Define a grammar called Hello
grammar Hello;
r  : 'hello' ID ;         // match keyword hello followed by an identifier
ID : [a-z]+ ;             // match lower-case identifiers
WS : [ \t\r\n]+ -> skip ; // skip spaces, tabs, newlines

The example (built from the standard Hello.g4 example):

input_ = antlr4.FileStream(_FILENAME)
lexer = HelloLexer.HelloLexer(input_)
stream = antlr4.CommonTokenStream(lexer)
parser = HelloParser.HelloParser(stream)

rule_name = 'r'
tree = getattr(parser, rule_name)()

I also wrote a listener. To assert/verify that this is correct, I'll repeat it here:

class HelloListener(antlr4.ParseTreeListener):
    def enterR(self, ctx):
        print("enterR")

    def exitR(self, ctx):
        print("exitR")

    def enterId(self, ctx):
        print("enterId")

    def exitId(self, ctx):
        print("exitId")

So, first, I can't guarantee that the string I'm giving it is valid because I'm not getting any screen output. How do I tell from the tree object if anything was matched? How do I extract the matching rules/tokens?

A Python example would be great, if possible.

Answer 1

I hear you, having the same issues right now. The Python documentation for v4 is useless and v3 differs to much to be usable. I'm thinking about switching back to Java to implement my stuff.

Regarding your code: I think your own custom listener has to inherit from the generated HelloListener. You can do the printing there.

Also try parsing invalid input to see if the parser starts at all. I'm not sure about the line with getattr(parser, rule_name)() though. I followed the steps in the (unfortunately very short) documentation for the Antlr4 Python target: https://theantlrguy.atlassian.net/wiki/display/ANTLR4/Python+Target

You can also find some documentation about the listener stuff there. Hope it helps.

Answer 2

This question seems to be old, but I also had the same problems and found out how to deal with it. When using strings in python, you have to use the function antlr4.InputStream as pointed out here

So, in the end, you could get a working example with this sort of code ( based on Alan's answer and an example from dzone )

from antlr4 import *
from grammar.HelloListener import HelloListener
from grammar.HelloLexer import HelloLexer
from grammar.HelloParser import HelloParser
import sys

class HelloPrintListener(HelloListener):
   def enterHi(self, ctx):
      print("Hello: %s" % ctx.ID())

def main():
   giveMeInput = input ("say hello XXXX\n")
   print("giveMeInput is {0}".format(giveMeInput))

   # https://www.programcreek.com/python/example/93166/antlr4.InputStream
   # https://groups.google.com/forum/#!msg/antlr-discussion/-9VJ5H9NcDs/OukVNCTQCAAJ
   i_stream = InputStream(giveMeInput)

   lexer = HelloLexer(i_stream)
   t_stream = CommonTokenStream(lexer)
   parser = HelloParser(t_stream)
   tree = parser.hi()
   printer = HelloPrintListener()
   walker = ParseTreeWalker()
   walker.walk(printer, tree)

if __name__ == '__main__':
   main()

Answer 3

I've created an example for Python 2 using the Hello grammar.

Here's the relevant code:

from antlr4 import *
from HelloLexer import HelloLexer
from HelloListener import HelloListener
from HelloParser import HelloParser
import sys

class HelloPrintListener(HelloListener):
    def enterHi(self, ctx):
        print("Hello: %s" % ctx.ID())

def main():
    lexer = HelloLexer(StdinStream())
    stream = CommonTokenStream(lexer)
    parser = HelloParser(stream)
    tree = parser.hi()
    printer = HelloPrintListener()
    walker = ParseTreeWalker()
    walker.walk(printer, tree)

if __name__ == '__main__':
    main()

As fabs said, the key is to inherit from the generated HelloListener. There seems to be some pickiness on this issue, as you can see if you modify my HelloPrintListener to inherit directly from ANTLR's ParseTreeListener . I imagined that would work since the generated HelloListener just has empty methods, but I saw the same behavior you saw (listener methods never being called).

Even though the documentation for Python listeners are lacking, the available methods are similar to Java.

Answer 4

The antlr documentation has been updated to document the support for python 3 and python 4 targets. The examples from the antlr book converted to python3 can be found here , they are sufficient enough to get anyone started.