Procedural macro parsing weirdness in Rust

Question

I'm trying to parse a macro similar to this one:

annoying!({
    hello({
        // some stuff
    });
})

Trying to do this with a procedural macro definition similar to the following, but I'm getting a behaviour I didn't expect and I'm not sure I'm doing something I'm not supposed to or I found a bug. In the following example, I'm trying to find the line where each block is, for the first block (the one just inside annoying!) it reports the correct line, but for the inner block, when I try to print them it's always 1, no matter where the code is etc.

#![crate_type="dylib"]
#![feature(macro_rules, plugin_registrar)]

extern crate syntax;
extern crate rustc;

use macro_result::MacroResult;
use rustc::plugin::Registry;
use syntax::ext::base::{ExtCtxt, MacResult};
use syntax::ext::quote::rt::ToTokens;
use syntax::codemap::Span;
use syntax::ast;
use syntax::parse::tts_to_parser;

mod macro_result;

#[plugin_registrar]
pub fn plugin_registrar(registry: &mut Registry) {
    registry.register_macro("annoying", macro_annoying);
}

pub fn macro_annoying(cx: &mut ExtCtxt, _: Span, tts: &[ast::TokenTree]) -> Box<MacResult> {
    let mut parser = cx.new_parser_from_tts(tts);

    let lo = cx.codemap().lookup_char_pos(parser.span.lo);
    let hi = cx.codemap().lookup_char_pos(parser.span.hi);

    println!("FIRST LO {}", lo.line); // real line for annoying! all cool
    println!("FIRST HI {}", hi.line); // real line for annoying! all cool

    let block_tokens = parser.parse_block().to_tokens(cx);
    let mut block_parser = tts_to_parser(cx.parse_sess(), block_tokens, cx.cfg());

    block_parser.bump(); // skip {
    block_parser.parse_ident(); // hello
    block_parser.bump(); // skip (

    // block lines
    let lo = cx.codemap().lookup_char_pos(block_parser.span.lo);
    let hi = cx.codemap().lookup_char_pos(block_parser.span.hi);

    println!("INNER LO {}", lo.line); // line 1? wtf?
    println!("INNER HI {}", hi.line); // line 1? wtf?

    MacroResult::new(vec![])
}

I think the problem might be the fact that I'm creating a second parser to parse the inner block, and that might be making the Span types inside it go crazy, but I'm not sure that's the problem or how to keep going from here. The reason I'm creating this second parser is so I can recursively parse what's inside each of the blocks, I might be doing something I'm not supposed to, in which case a better suggestion would be very welcome.

Answer 1

I believe this is #15962 (and #16472 ), to_tokens has a generally horrible implementation. Specifically, anything non-trivial uses ToSource , which just turns the code to a string, and then retokenises that (yes, it's not great at all!).

Until those issues are fixed, you should just handle the original tts directly as much as possible. You could approximate the right span using the .span of the parsed block (ie return value of parse_block ), which will at least focus the user's attention on the right area.