Anonymous vs struct lifetime for assignment

Question

For this code (trimmed some, sorry not more), I get a lifetime problem:

fn main() {
    println!("Hello, world!");
}

#[derive(Debug)]
pub struct Token<'a> {
    pub line: usize,
    // Col in code points.
    pub col: usize,
    // Index in bytes.
    pub index: usize,
    pub state: TokenState,
    pub text: &'a str,
}

#[derive(Clone, Copy, Debug, PartialEq)]
pub enum TokenState {
    VSpace,
}

#[derive(Clone, Copy, Debug, PartialEq)]
pub enum ParseState {
    Expr,
}

pub struct Node<'a> {
    kids: Vec<Node<'a>>,
    state: ParseState,
    token: Option<&'a Token<'a>>,
}

impl<'a> Node<'a> {
    fn new(state: ParseState) -> Node<'a> {
        Node {
            kids: vec![],
            state,
            token: None,
        }
    }

    fn new_token(token: &'a Token<'a>) -> Node<'a> {
        // TODO Control state? Some token state?
        Node {
            kids: vec![],
            state: ParseState::Expr,
            token: Some(&token),
        }
    }

    fn push_if(&mut self, node: Node<'a>) {
        if !node.kids.is_empty() {
            self.kids.push(node);
        }
    }
}

pub fn parse<'a>(tokens: &'a Vec<Token<'a>>) -> Node<'a> {
    let mut root = Node::new(ParseState::Expr);
    let mut parser = Parser {
        index: 0,
        tokens: tokens,
    };
    parser.parse_block(&mut root);
    root
}

struct Parser<'a> {
    index: usize,
    tokens: &'a Vec<Token<'a>>,
}

impl<'a> Parser<'a> {
    fn parse_block(&mut self, parent: &mut Node) {
        loop {
            let mut row = Node::new(ParseState::Expr);
            match self.peek() {
                Some(_) => {
                    self.parse_row(&mut row);
                }
                None => {
                    break;
                }
            }
            parent.push_if(row);
        }
    }

    fn parse_row(&mut self, parent: &mut Node) {
        loop {
            match self.next() {
                Some(ref token) => match token.state {
                    TokenState::VSpace => break,
                    _ => {
                        parent.kids.push(Node::new_token(&token));
                    }
                },
                None => break,
            }
        }
    }

    fn next(&mut self) -> Option<&Token> {
        let index = self.index;
        if index < self.tokens.len() {
            self.index += 1;
        }
        self.tokens.get(index)
    }

    fn peek(&mut self) -> Option<&Token> {
        self.tokens.get(self.index)
    }
}

( playground )

This is the error message:

error[E0495]: cannot infer an appropriate lifetime for autoref due to conflicting requirements
  --> src/main.rs:90:24
   |
90 |             match self.next() {
   |                        ^^^^
   |
note: first, the lifetime cannot outlive the lifetime 'a as defined on the impl at 72:1...
  --> src/main.rs:72:1
   |
72 | / impl<'a> Parser<'a> {
73 | |     fn parse_block(&mut self, parent: &mut Node) {
74 | |         loop {
75 | |             let mut row = Node::new(ParseState::Expr);
...  |
112| |     }
113| | }
   | |_^
note: ...so that the type `Parser<'a>` is not borrowed for too long
  --> src/main.rs:90:19
   |
90 |             match self.next() {
   |                   ^^^^
note: but, the lifetime must be valid for the anonymous lifetime #3 defined on the method body at 88:5...
  --> src/main.rs:88:5
   |
88 | /     fn parse_row(&mut self, parent: &mut Node) {
89 | |         loop {
90 | |             match self.next() {
91 | |                 Some(ref token) => match token.state {
...  |
99 | |         }
100| |     }
   | |_____^
note: ...so that expression is assignable (expected Node<'_>, found Node<'_>)
  --> src/main.rs:94:42
   |
94 |                         parent.kids.push(Node::new_token(&token));
   |                                          ^^^^^^^^^^^^^^^^^^^^^^^

All the references should be tied to the same outside lifetime. In my full code (of which I just have an excerpt here), I expect to hang onto the original parsed source, and I'm trying to tie everything to that.

I know the error messages are trying to be helpful, but I'm really unsure what the conflict is. And I'm unsure what other lifetime questions here are related to the same issue I have or not.

Answer 1

Let's take a look at the signature of Parser::next :

fn next(&mut self) -> Option<&Token>

This function promises to return an Option<&Token> . There are elided lifetimes here; let's rewrite the signature to make them explicit:

fn next<'b>(&'b mut self) -> Option<&'b Token<'b>>

We can now see that next is generic over lifetime 'b . Notice how the return type uses 'b , not 'a . This is valid in itself, because the compiler can infer that 'b is a shorter than 'a , and mutable references ( &'a mut T ) are covariant over 'a ("covariant" in this context means that we can substitute lifetime 'a with a shorter lifetime). But what the function ends up promising is that the result lives at least as long as itself, while it can in fact live at least as long as 'a .

In Parser::parse_row , you're trying to take the result of Parser::next and insert it into parent . Let's look at Parser::parse_row 's signature:

fn parse_row(&mut self, parent: &mut Node)

We have some omitted lifetimes here again. Let's spell them out:

fn parse_row<'b, 'c, 'd>(&'b mut self, parent: &'c mut Node<'d>)

'c is not going to be important, so we can ignore it.

If we try to compile now, the last two notes are different:

note: but, the lifetime must be valid for the lifetime 'd as defined on the method body at 88:5...
  --> src/main.rs:88:5
   |
88 | /     fn parse_row<'b, 'c, 'd>(&'b mut self, parent: &'c mut Node<'d>) {
89 | |         loop {
90 | |             match self.next() {
91 | |                 Some(ref token) => match token.state {
...  |
99 | |         }
100| |     }
   | |_____^
note: ...so that expression is assignable (expected Node<'d>, found Node<'_>)
  --> src/main.rs:94:42
   |
94 |                         parent.kids.push(Node::new_token(&token));
   |                                          ^^^^^^^^^^^^^^^^^^^^^^^

Now, one of the anonymous lifetimes is identified as 'd . The other is still an anonymous lifetime, and that's an artifact of how the compiler manipulates lifetimes, but we can think of it as being 'b here.

The problem should be a bit clearer now: we're trying to push a Node<'b> into a collection of Node<'d> objects. It's important that the type be exactly Node<'d> , because mutable references ( &'a mut T ) are invariant over T ("invariant" means it can't change).

Let's make the lifetimes match. First, we'll change next 's signature to match what we can actually return:

fn next(&mut self) -> Option<&'a Token<'a>>

This means that now, when we call self.next() in parse_row , we'll be able to construct a Node<'a> . A Node<'x> can only store Node<'x> objects (per your definition of Node ), so the parent parameter's referent must also be of type Node<'a> .

fn parse_row(&mut self, parent: &mut Node<'a>)

If we try to compile now, we'll get an error in Parser::parse_block on the call to parse_row . The problem is similar to what we just saw. parse_block 's signature is:

fn parse_block(&mut self, parent: &mut Node)

which expands to:

fn parse_block<'b, 'c, 'd>(&'b mut self, parent: &'c mut Node<'d>)

Here's the error the compiler gives with this elaborated signature:

error[E0495]: cannot infer an appropriate lifetime for lifetime parameter `'a` due to conflicting requirements
  --> src/main.rs:78:26
   |
78 |                     self.parse_row(&mut row);
   |                          ^^^^^^^^^
   |
note: first, the lifetime cannot outlive the lifetime 'a as defined on the impl at 72:1...
  --> src/main.rs:72:1
   |
72 | / impl<'a> Parser<'a> {
73 | |     fn parse_block<'b, 'c, 'd>(&'b mut self, parent: &'c mut Node<'d>) {
74 | |         loop {
75 | |             let mut row = Node::new(ParseState::Expr);
...  |
112| |     }
113| | }
   | |_^
note: ...so that types are compatible (expected &mut Parser<'_>, found &mut Parser<'a>)
  --> src/main.rs:78:26
   |
78 |                     self.parse_row(&mut row);
   |                          ^^^^^^^^^
note: but, the lifetime must be valid for the lifetime 'd as defined on the method body at 73:5...
  --> src/main.rs:73:5
   |
73 | /     fn parse_block<'b, 'c, 'd>(&'b mut self, parent: &'c mut Node<'d>) {
74 | |         loop {
75 | |             let mut row = Node::new(ParseState::Expr);
76 | |             match self.peek() {
...  |
85 | |         }
86 | |     }
   | |_____^
note: ...so that types are compatible (expected &mut Node<'_>, found &mut Node<'d>)
  --> src/main.rs:84:20
   |
84 |             parent.push_if(row);
   |                    ^^^^^^^

The compiler is unable to infer the type of row (specifically, the lifetime in its type Node<'x> ). On one hand, the call to parse_row means it should be Node<'a> , but the call to push_if means it should be Node<'d> . 'a and 'd are unrelated, so the compiler doesn't know how to unify them.

The solution is easy, and it's the same as above: just make parent have type &mut Node<'a> .

fn parse_block(&mut self, parent: &mut Node<'a>)

Now your code compiles!