如何使用Lark解析无效的JSON？

Question

Let's start by considering a simple json parser using lark:

import sys

from lark import Lark, Transformer, v_args

json_grammar = r"""
    ?start: value

    ?value: object
          | array
          | string
          | SIGNED_NUMBER      -> number
          | "true"             -> true
          | "false"            -> false
          | "null"             -> null

    array  : "[" [value ("," value)*] "]"
    object : "{" [pair ("," pair)*] "}"
    pair   : string ":" value

    string : ESCAPED_STRING

    %import common.ESCAPED_STRING
    %import common.SIGNED_NUMBER
    %import common.WS

    %ignore WS
"""


class TreeToJson(Transformer):
    @v_args(inline=True)
    def string(self, s):
        return s[1:-1].replace('\\"', '"')

    array = list
    pair = tuple
    object = dict
    number = v_args(inline=True)(float)

    def null(self, _): return None

    def true(self, _): return True

    def false(self, _): return False


if __name__ == '__main__':
    json_parser = Lark(json_grammar, parser='lalr', lexer='standard', transformer=TreeToJson())
    parse = json_parser.parse

    dct = parse('''
        {
            "empty_object" : {},
            "empty_array"  : [],
            "booleans"     : { "YES" : true, "NO" : false },
            "numbers"      : [ 0, 1, -2, 3.3, 4.4e5, 6.6e-7 ],
            "strings"      : [ "This", [ "And" , "That", "And a \\"b" ] ],
            "nothing"      : null
        }
    ''')
    print(dct)

The above example is taken from the official examples website and it's able to parse valid json.

So far so good but my question would be how I could extend this grammar & transformer so it will also be able to parse invalid json strings such as the below one:

dct = parse('''
    [
        // Item1
        { "key1": "value1" },
        // Item2
        { "key2": "value2", "key3": ["a","b",] },
        // Item3
        { "key4": [{"key5":"value5"},] },
    ]
''')

My main goal is to be able of parsing SublimeText assets (which are a superset of json), ST uses sublime_api.decode_value behind the curtains... but this function is closed source so I can't use it. Also I didn't find any pypi library which works out of the box for this type of data so I decided my best chance would trying to write my own custom "invalid json" parser.

Answer 1

The demjson library is very good at parsing questionable json:

import demjson

str = '''
    [
        // Item1
        { "key1": "value1" },
        // Item2
        { "key2": "value2", "key3": ["a","b",] },
        // Item3
        { "key4": [{"key5":"value5"},] },
    ]
'''

print(demjson.decode(str))

Result:

[{'key1': 'value1'}, {'key2': 'value2', 'key3': ['a', 'b']}, {'key4': [{'key5': 'value5'}]}]