Confusion around priority of tokens in lark grammar

Question

Following up from an earlier question , I'm a bit confused about the precedence of the /.+/ regex line; I would expect the below test to produce

  line
  line  x
  chunk abc

instead I get:

  line
  line  x
  line  abc

    def test_tokenizing(self):
        p = Lark(r"""

        _NL: /\n/
        line.-1: /.+/? _NL
        chunk: /abc/ _NL
        start: (line|chunk)+

        """, parser='lalr')

        text = '\nx\nabc\n'
        print(p.parse(text).pretty())

Answer 1

In Lark, priorities mean different things for rules and for terminals.

Just a quick reminder, rules have lowercase names, while terminals have UPPERCASE names.

In LALR mode, priorities on rules only affect which one is chosen in case of a reduce/reduce collision. It has no effect on the terminals inside it.

What you want is to change the priority on the terminal itself:

def test_tokenizing():
    p = Lark(r"""

    _NL: /\n/
    line: EVERYTHING? _NL
    EVERYTHING.-1: /.+/
    chunk: /abc/ _NL
    start: (line|chunk)+

    """, parser='lalr')

    text = '\nx\nabc\n'
    print(p.parse(text).pretty())