如何编辑解析器以正确组合“ AND”和“ OR”谓词？

Question

I am currently trying to write a small parser able to parse very simple key = value queries. But it should be smart enough to handle AND and OR groups, with AND having a higher precendence. Example text-input:

a = 10 && b = 20
a = 10 || b = 20
a = 10 && b = 20 || c = 30

The first two are trivial. The last should group the first two predicates as an "AND" group, and that group should then be grouped in an "OR" group.

I have the basics down, but got stuck on the proper grouping. I am using ply which uses a flex/bison/lex/yacc syntax to define the grammar. If I'm totally heading down a wrong track with my existing syntax please let me know... That would be a valuable learning experience concerning parsers.

I've tried setting the precedence, but I don't think it's really caused by a reduce/reduce conflict. I think it's more of an issue of the way I've defined my grammar in general, but I can't figure out what I need to change.

Below is my current implementation and a unit-test file. The test-file should help understanding the expected output. There's currently one failing test. That's the one that causes me headaches.

The tests can be run using the builtin unittest module, but, as I execute some print statements in the tests, I suggest using pytest as it intercepts those and causes less of a mess. For example (assuming both files are in the same folder):

python -m venv env
./env/bin/pip install pytest
./env/bin/pytest test_query_string.py

---

Filename: queryparser.py

import logging
from collections import namedtuple

import ply.lex as lex
import ply.yacc as yacc


LOG = logging.getLogger(__name__)

Predicate = namedtuple('Predicate', 'key operator value')


class Production:

    def __repr__(self):
        preds = [repr(pred) for pred in self._predicates]
        return '%s(%s)' % (self.__class__.__name__, ', '.join(preds))

    def __eq__(self, other):
        return (
            self.__class__ == other.__class__ and
            self._predicates == other._predicates)

    def debug(self, indent=0, aslist=False):
        lines = []
        lines.append('    ' * indent + self.__class__.__name__)
        for predicate in self._predicates:
            if hasattr(predicate, 'debug'):
                lines.extend(predicate.debug(indent + 1, aslist=True))
            else:
                lines.append('    ' * (indent+1) + repr(predicate))
        if aslist:
            return lines
        else:
            return '\n'.join(lines)



class Conjunction(Production):

    def __init__(self, *predicates):
        self._predicates = predicates


class Disjunction(Production):

    def __init__(self, *predicates):
        self._predicates = predicates


def parse(query: str, debug=False) -> Predicate:
    lexer = QueryLexer().build()
    parser = QueryParser().build()
    if debug:
        output = parser.parse(query, debug=LOG)
    else:
        output = parser.parse(query)
    return output or []


class QueryLexer:

    tokens = (
        'WORD',
        'OPERATOR',
        'QUOTE',
        'AND',
        'OR'
    )

    t_ignore = ' \t'
    t_QUOTE = '"'

    def t_error(self, t):
        LOG.warning('Illegal character %r', t.value[0])
        t.lexer.skip(1)

    def t_WORD(self, t):
        r'\w+'
        return t

    def t_OPERATOR(self, t):
        r'(=|!=|>|<|<=|>=)'
        return t

    def t_AND(self, t):
        r'&&'
        return t

    def t_OR(self, t):
        r'\|\|'
        return t

    def build(self, **kwargs):
        self.lexer = lex.lex(module=self, **kwargs)


class QueryParser:

    precedence = (
        ('nonassoc', 'OR'),
        ('nonassoc', 'AND'),
    )

    def p_query(self, p):
        '''
        query : disjunction
              | conjunction
              | predicate
        '''
        p[0] = p[1]

    def p_disjunction(self, p):
        '''
        disjunction : predicate OR predicate
                    | predicate OR conjunction
                    | predicate OR disjunction
        '''
        output = [p[1]]
        if p.slice[3].type == 'disjunction':
            # We can merge multiple chanined disjunctions together
            output.extend(p[3]._predicates)
        else:
            output.append(p[3])
        p[0] = Disjunction(*output)

    def p_conjunction(self, p):
        '''
        conjunction : predicate AND predicate
                    | predicate AND conjunction
                    | predicate AND disjunction
        '''
        if len(p) == 4:
            output = [p[1]]
            if p.slice[3].type == 'conjunction':
                # We can merge multiple chanined disjunctions together
                output.extend(p[3]._predicates)
            else:
                output.append(p[3])
            p[0] = Conjunction(*output)
        else:
            p[0] = Conjunction(p[1])

    def p_predicate(self, p):
        '''
        predicate : maybequoted OPERATOR maybequoted
        '''
        p[0] = Predicate(p[1], p[2], p[3])

    def p_maybequoted(self, p):
        '''
        maybequoted : WORD
                    | QUOTE WORD QUOTE
        '''
        if len(p) == 4:
            p[0] = p[2]
        else:
            p[0] = p[1]

    def p_error(self, p):
        """
        Panic-mode rule for parser errors.
        """
        if not p:
            LOG.debug('Syntax error at EOF')
        else:
            self.parser.errok()
        LOG.error('Syntax Error at %r', p)

    def build(self):
        self.tokens = QueryLexer.tokens
        self.parser = yacc.yacc(module=self, outputdir='/tmp', debug=True)
        return self.parser

---

Filename: test_query_string.py

import unittest

from queryparser import parse, Conjunction, Disjunction, Predicate


class TestQueryString(unittest.TestCase):

    def test_single_equals(self):
        result = parse('hostname = foo')
        self.assertEqual(result, Predicate('hostname', '=', 'foo'))

    def test_single_equals_quoted(self):
        result = parse('hostname = "foo"')
        self.assertEqual(result, Predicate('hostname', '=', 'foo'))

    def test_anded_equals(self):
        result = parse('hostname = foo && role=cpe')
        self.assertEqual(result, Conjunction(
            Predicate('hostname', '=', 'foo'),
            Predicate('role', '=', 'cpe'),
        ))

    def test_ored_equals(self):
        result = parse('hostname = foo || role=cpe')
        self.assertEqual(result, Disjunction(
            Predicate('hostname', '=', 'foo'),
            Predicate('role', '=', 'cpe'),
        ))

    def test_chained_conjunction(self):
        result = parse('hostname = foo && role=cpe && bla=blub')
        print(result.debug())  # XXX debug statement
        self.assertEqual(result, Conjunction(
            Predicate('hostname', '=', 'foo'),
            Predicate('role', '=', 'cpe'),
            Predicate('bla', '=', 'blub'),
        ))

    def test_chained_disjunction(self):
        result = parse('hostname = foo || role=cpe || bla=blub')
        print(result.debug())  # XXX debug statement
        self.assertEqual(result, Disjunction(
            Predicate('hostname', '=', 'foo'),
            Predicate('role', '=', 'cpe'),
            Predicate('bla', '=', 'blub'),
        ))

    def test_mixed_predicates(self):
        result = parse('hostname = foo || role=cpe && bla=blub')
        print(result.debug())  # XXX debug statement
        self.assertEqual(result, Disjunction(
            Predicate('hostname', '=', 'foo'),
            Conjunction(
                Predicate('role', '=', 'cpe'),
                Predicate('bla', '=', 'blub'),
            )
        ))

    def test_mixed_predicate_and_first(self):
        result = parse('hostname = foo && role=cpe || bla=blub')
        print(result.debug())  # XXX debug statement
        self.assertEqual(result, Conjunction(
            Predicate('hostname', '=', 'foo'),
            Disjunction(
                Predicate('role', '=', 'cpe'),
                Predicate('bla', '=', 'blub'),
            )
        ))

    def test_complex(self):
        result = parse(
            'a=1 && b=2 || c=3 && d=4 || e=5 || f=6 && g=7 && h=8',
            debug=True
        )
        print(result.debug())  # XXX debug statement

        expected = Disjunction(
            Conjunction(
                Predicate('a', '=', '1'),
                Predicate('b', '=', '2'),
            ),
            Conjunction(
                Predicate('c', '=', '3'),
                Predicate('d', '=', '4'),
            ),
            Predicate('e', '=', '5'),
            Conjunction(
                Predicate('f', '=', '6'),
                Predicate('g', '=', '7'),
                Predicate('h', '=', '8'),
            ),
        )

        self.assertEqual(result, expected)

Answer 1

If you are using precedence declarations, both AND and OR should be declared as left , not nonassoc . nonassoc means that a OR b OR c is illegal; left means that it is to be interpreted as (a OR b) OR c) and right means a OR (b OR c) . (Given the semantics of AND and OR , it makes no difference whether left or right is chosen, but left is generally preferable in such cases.)

With precedence relationships, it is possible to write an extremely simple grammar:

query: predicate
     | query AND query
     | query OR query

(Usually, there would also be an entry for parenthesized expressions.)

The above does not do the chaining you are looking for. You could do that post-parse by walking the tree, which would generally be my preference. But it also is possible to chain on the fly, using a grammar with explicit precedence.

Explicit precedence means that the grammar itself defines what it possible; in particular, since AND binds more tightly than OR , it is not possible to have conjunction: predicate AND disjunction precisely because that production implies that the second operand to AND could be a disjunction, which is not the desired outcome. For this case, you want the common cascading sequence:

query       : disjunction  # Redundant, but possibly useful for didactic purposes
disjunction : conjunction
            | disjunction OR conjunction   # Left associative
conjunction : predicate
            | conjunction AND predicate

With that grammar, chaining is straight-forward, but it requires an explicit test as in your actions (eg., if p.slice(1).type == 'conjunction: ) which is arguably a bit ugly.

Ideally, we would want to trigger the correct action directly from the grammar, which would imply something like this (which is very similar to your grammar):

conjunction: predicate
                # p[0] = p[1]
           | predicate AND predicate
                # p[0] = Conjunction(p[1], p[3])
           | conjunction AND predicate
                # p[0] = Conjunction(*(p[1]._predicates + [p[3]])

The problem with the above rules is that the second and the third both apply to a AND b , since after reducing a to predicate we have both the option to reduce it to conjunction or to shift the AND immediately. In this case, we want the parser to resolve the shift-reduce conflict by unconditionally shifting, which it will do, but only after producing a warning. For an explicit solution, we need the conjunction in the third rule to be a real conjunction, with at least one AND operator.

With that in mind, we can shift the unit productions to the top of the cascade, resulting in the following:

query      : disjunction
           | conjunction
           | predicate
disjunction: predicate OR predicate
           | conjunction OR predicate
           | disjunction OR predicate
conjunction: predicate AND predicate
           | conjunction AND predicate

Now we have no need for conditionals in the actions, because we know exactly what we have in every case.

def p_query(self, p):
    '''
    query : disjunction
          | conjunction
          | predicate
    '''
    p[0] = p[1]

def p_disjunction1(self, p):
    '''
    disjunction: predicate OR predicate
               | conjunction OR predicate
    '''
    p[0] = Disjunction(p[1], p[3])

def p_disjunction2(self, p):
    '''
    disjunction: disjunction OR predicate
    '''
    p[0] = Disjunction(*(p[1]._predicate + [p[3]])

def p_conjunction1(self, p):
    '''
    conjunction: predicate AND predicate
    '''
    p[0] = Conjunction(p[1], p[3])

def p_conjunction2(self, p):
    '''
    conjunction: conjunction AND predicate
    '''
    p[0] = Disjunction(*(p[1]._predicate + [p[3]])

---

Notes

1. The grammar provided is fine for the case of two precedence levels, but the number of productions ends up being quadratic in the number of levels. If that is annoying, an alternative model with more unit productions:

    query : disjunction disjunction : conjunction | disjunction_2 disjunction_2 : conjunction OR predicate | disjunction_2 OR predicate conjunction : predicate | conjunction_2 conjunction_2 : predicate AND predicate | conjunction_2 AND predicate 

2. If you don't insist on parser objects being immutable, you could combine both of the chaining functions ( p_conjunction2 and p_disjunction2 ) into a single function:

    def p_chain(self, p): ''' conjunction: conjunction AND predicate disjunction: disjunction OR predicate ''' p[0] = p[1] p[0]._predicate.append(p[3]) 

   Additional simplification could be achieved by making the value of the operator tokens AND and OR the constructor instead of the matched string. (The matched string is really redundant, anyway.) This would allow the constructor functions ( p_disjunction1 and p_conjunction1 to also be replaced with a single function:

    def t_AND(self, t): r'&&' t.value = Conjunction return t def t_OR(self, t): r'\\|\\|' t.value = Disjunction return t # ... def p_construct(self, p): ''' disjunction: predicate OR predicate | conjunction OR predicate conjunction: predicate AND predicate ''' p[0] = p[2](p[1], p[3])