Python3 Antlr4 AttributeError: 'CommonToken' 对象没有属性 'getLine'

Question

I'm using Antlr4 with Python3.

I have a getTerminalPos method which returns a tuple of the line and column number of a given terminal. To do this, I first get the token using .getSymbol() and I then use the .getLine() and .getCharPositionInLine() methods to get the positions.

def getTerminalPos(self, terminal):
  token = terminal.getSymbol()
  return (token.getLine(), token.getCharPositionInLine())

An example of calling getTerminalPos inside an antlr visitor:

def visitAtom(self, ctx):
  if ctx.LPAREN():
    return self.visitExpr(ctx.expr())

  if ctx.INT():
    return nodes.Number(int(ctx.INT().getText()), getTerminalPos(ctx.INT()))

  if ctx.FLOAT():
    return nodes.Number(float(ctx.FLOAT().getText()), getTerminalPos(ctx.FLOAT()))

When I run the code, I get the following error message:

  File ".../py-antlr4-lmaspl/AntlrVisitor.py", line 55, in getTerminalPos
    return (token.getLine(), token.getCharPositionInLine())
AttributeError: 'CommonToken' object has no attribute 'getLine'

According to the Antlr4 Java Runtime, these methods exist: https://www.antlr.org/api/Java/org/antlr/v4/runtime/CommonToken.html

According to the Antlr3 Python Runtime, these methods exist: https://www.antlr3.org/api/Python/classantlr3_1_1_common_token.html

So, they should exist for the Antlr4 Python Runtime too?

How do I fix this error? Is there a different set of methods I should use instead to get the line and column numbers?

Edit : I meant to say that I found a similar issue here: https://github.com/antlr/antlr4/issues/1529 . It is marked as a bug, but closed for now...

Answer 1

If I look at the source of the Python 3 runtime, I see CommonToken like this :

class CommonToken(Token):

    # An empty {@link Pair} which is used as the default value of
    # {@link #source} for tokens that do not have a source.
    EMPTY_SOURCE = (None, None)

    def __init__(self, source = EMPTY_SOURCE, type = None, channel=Token.DEFAULT_CHANNEL, start=-1, stop=-1):
        super(CommonToken, self).__init__()
        self.source = source
        self.type = type
        self.channel = channel
        self.start = start
        self.stop = stop
        self.tokenIndex = -1
        if source[0] is not None:
            self.line = source[0].line
            self.column = source[0].column
        else:
            self.column = -1

    ...

and Token like this:

class Token (object):

    ...

    def __init__(self):
        self.source = None
        self.type = None # token type of the token
        self.channel = None # The parser ignores everything not on DEFAULT_CHANNEL
        self.start = None # optional; return -1 if not implemented.
        self.stop = None  # optional; return -1 if not implemented.
        self.tokenIndex = None # from 0..n-1 of the token object in the input stream
        self.line = None # line=1..n of the 1st character
        self.column = None # beginning of the line at which it occurs, 0..n-1
        self._text = None # text of the token.

    ...

So, my guess is this should do it for you:

return (token.line, token.column)