How do I get the value and type of a Literal in a procedural macro?

Question

I am implementing a function-like procedural macro which takes a single string literal as an argument, but I don't know how to get the value of the string literal.

If I print the variable, it shows a bunch of fields, which includes both the type and the value. They are clearly there, somewhere. How do I get them?

extern crate proc_macro;
use proc_macro::{TokenStream,TokenTree};

#[proc_macro]
pub fn my_macro(input: TokenStream) -> TokenStream {
    let input: Vec<TokenTree> = input.into_iter().collect();
    let literal = match &input.get(0) {
        Some(TokenTree::Literal(literal)) => literal,
        _ => panic!()
    };

    // can't do anything with "literal"
    // println!("{:?}", literal.lit.symbol); says "unknown field"

    format!("{:?}", format!("{:?}", literal)).parse().unwrap()
}

#![feature(proc_macro_hygiene)]
extern crate macros;

fn main() {
    let value = macros::my_macro!("hahaha");
    println!("it is {}", value);
    // prints "it is Literal { lit: Lit { kind: Str, symbol: "hahaha", suffix: None }, span: Span { lo: BytePos(100), hi: BytePos(108), ctxt: #0 } }"
}

Answer 1

If you're writing procedural macros, I'd recommend that you look into using the crates syn (for parsing) and quote (for code generation) instead of using proc-macro directly, since those are generally easier to deal with.

In this case, you can use syn::parse_macro_input to parse a token stream into any syntatic element of Rust (such as literals, expressions, functions), and will also take care of error messages in case parsing fails.

You can use LitStr to represent a string literal, if that's exactly what you need. The .value() function will give you a String with the contents of that literal.

You can use quote::quote to generate the output of the macro, and use # to insert the contents of a variable into the generated code.

use proc_macro::TokenStream;
use syn::{parse_macro_input, LitStr};
use quote::quote;

#[proc_macro]
pub fn my_macro(input: TokenStream) -> TokenStream {
    // macro input must be `LitStr`, which is a string literal.
    // if not, a relevant error message will be generated.
    let input = parse_macro_input!(input as LitStr);

    // get value of the string literal.
    let str_value = input.value();

    // do something with value...
    let str_value = str_value.to_uppercase();

    // generate code, include `str_value` variable (automatically encodes
    // `String` as a string literal in the generated code)
    (quote!{
        #str_value
    }).into()
}

Answer 2

After running into the same problem countless times already, I finally wrote a library to help with this: litrs on crates.io . It compiles faster than syn and lets you inspect your literals.

use std::convert::TryFrom;
use litrs::StringLit;
use proc_macro::TokenStream;
use quote::quote;


#[proc_macro]
pub fn my_macro(input: TokenStream) -> TokenStream {
    let input = input.into_iter().collect::<Vec<_>>();
    if input.len() != 1 {
        let msg = format!("expected exactly one input token, got {}", input.len());
        return quote! { compile_error!(#msg) }.into();
    }

    let string_lit = match StringLit::try_from(&input[0]) {
        // Error if the token is not a string literal
        Err(e) => return e.to_compile_error(),
        Ok(lit) => lit,
    };

    // `StringLit::value` returns the actual string value represented by the
    // literal. Quotes are removed and escape sequences replaced with the
    // corresponding value.
    let v = string_lit.value();

    // TODO: implement your logic here
}

See the documentation of litrs for more information.

---

To obtain more information about a literal, litrs uses the Display impl of Literal to obtain a string representation (as it would be written in source code) and then parses the that string. For example, if the string starts with 0x one knows it has to be an integer literal, if it starts with r#" one knows it is a raw string literal. The crate syn does exactly the same.

Of course, it seems a bit wasteful to write and run a second parser given that rustc already parsed the literal. Yes, that's unfortunate and having a better API in proc_literal would be preferable. But right now, I think litrs (or syn if you are using syn anyway) are the best solutions.

---

(PS: I'm usually not a fan of promoting one's own libraries on Stack Overflow, but I am very familiar with the problem OP is having and I very much think litrs is the best tool for the job right now.)

Answer 3

I always want a string literal, so I found this solution that is good enough. Literal implements ToString , which I can then use with .parse() .

#[proc_macro]
pub fn my_macro(input: TokenStream) -> TokenStream {
    let input: Vec<TokenTree> = input.into_iter().collect();
    let value = match &input.get(0) {
        Some(TokenTree::Literal(literal)) => literal.to_string(),
        _ => panic!()
    };
    let str_value: String = value.parse().unwrap();

    // do whatever

    format!("{:?}", str_value).parse().unwrap()
}

Answer 4

I had similar problem for parsing doc attribute. It is also represented as a TokenStream . This is not exact answer but maybe will guide in a proper direction:

fn from(value: &Vec<Attribute>) -> Vec<String> {
    let mut lines = Vec::new();
    for attr in value {
        if !attr.path.is_ident("doc") {
            continue;
        }
        if let Ok(Meta::NameValue(nv)) = attr.parse_meta() {
            if let Lit::Str(lit) = nv.lit {
                lines.push(lit.value());
            }
        }
    }
    lines
}