使用解析器组合器解析带有转义字符的字符串？

Question

I'm trying to use the combine library in Rust to parse a string. The real data that I'm trying to parse looks something like this:

A79,216,0,4,2,2,N,"US\"PS"

So at the end of that data is a string in quotes, but the string will contain escaped characters as well. I can't figure out how to parse those escaped characters in between the other quotes.

extern crate parser_combinators;

use self::parser_combinators::*;

fn main() {
    let s = r#""HE\"LLO""#;
    let data = many(satisfy(|c| c != '"')); // Fails on escaped " obviously
    let mut str_parser = between(satisfy(|c| c == '"'), satisfy(|c| c == '"'), data);
    let result : Result<(String, &str), ParseError> = str_parser.parse(s);
    match result {
        Ok((value, _)) => println!("{:?}", value),
        Err(err) => println!("{}", err),
    }
}

//=> "HE\\"

The code above will parse that string successfully but will obviously fail on the escaped character in the middle, printing out "HE\\\\" in the end.

I want to change the code above so that it prints "HE\\\\\\"LLO" .

How do I do that?

Answer 1

I have a mostly functional JSON parser as a benchmark for parser-combinators which parses this sort of escaped characters. I have included a link to it and a slightly simplified version of it below.

fn json_char(input: State<&str>) -> ParseResult<char, &str> {
    let (c, input) = try!(satisfy(|c| c != '"').parse_state(input));
    let mut back_slash_char = satisfy(|c| "\"\\nrt".chars().find(|x| *x == c).is_some()).map(|c| {
        match c {
            '"' => '"',
            '\\' => '\\',
            'n' => '\n',
            'r' => '\r',
            't' => '\t',
            c => c//Should never happen
        }
    });
    match c {
        '\\' => input.combine(|input| back_slash_char.parse_state(input)),
        _    => Ok((c, input))
    }
}

json_char

Since this parser may consume 1 or 2 characters it is not enough to use the primitive combinators and so we need to introduce a function which can branch on the character which is parsed.

Answer 2

I ran into the same problem and ended up with the following solution:

    (
        char('"'),
        many1::<Vec<char>, _>(choice((
            escaped_character(),
            satisfy(|c| c != '"'),
        ))),
        char('"')
    )

Or in other words, a string is delimited by " followed by many escaped_characters or anything that isn't a closing " , and is closed by a closing " .

Here's a full example of how I'm using this:

pub enum Operand {
    String { value: String },
}

fn escaped_character<I>() -> impl Parser<Input = I, Output = char>
    where
        I: Stream<Item = char>,
        I::Error: ParseError<I::Item, I::Range, I::Position>,
{
    (
        char('\\'),
        any(),
    ).and_then(|(_, x)| match x {
        '0' => Ok('\0'),
        'n' => Ok('\n'),
        '\\' => Ok('\\'),
        '"' => Ok('"'),
        _ => Err(StreamErrorFor::<I>::unexpected_message(format!("Invalid escape sequence \\{}", x)))
    })
}

#[test]
fn parse_escaped_character() {
    let expected = Ok(('\n', " foo"));
    assert_eq!(expected, escaped_character().easy_parse("\\n foo"))
}

fn string_operand<I>() -> impl Parser<Input = I, Output = Operand>
    where
        I: Stream<Item = char>,
        I::Error: ParseError<I::Item, I::Range, I::Position>,
{
    (
        char('"'),
        many1::<Vec<char>, _>(choice((
            escaped_character(),
            satisfy(|c| c != '"'),
        ))),
        char('"')
    )
        .map(|(_,value,_)| Operand::String { value: value.into_iter().collect() })
}

#[test]
fn parse_string_operand() {
    let expected = Ok((Operand::String { value: "foo \" bar \n baz \0".into() }, ""));
    assert_eq!(expected, string_operand().easy_parse(r#""foo \" bar \n baz \0""#))
}