如何在我自己的解析器中使用flex？

Question

I want to leave the lexical analysis to lex but develop the parser on my own.

I made a token.h header which has the enums for token types and a simple class hierarchy,

For the lex rule:

[0-9]+ {yylval = new NumToken(std::stoi(yytext));return NUM;}

How do I get the NumToken pointer from the parser code? Suppose I just want to print out the tokens..

while(true)
{
    auto t = yylex();
    //std::cout <<yylval.data<<std::endl; // What goes here ?
}

I can do this with yacc/bison, but can not find any documentation or example about how to do this manually.

Answer 1

In a traditional bison/flex parser, yylval is a global variable defined in the parser generated by bison, and declared in the header file generated by bison (which should be #include'd into the generated scanner). So a simple solution would be just to replicate that: declare yylval (as a global) in token.h and define it somewhere in your parser.

But modern programming style has shifted away from the use of globals (for good reason), and indeed even flex will generate scanners which do not depend on global state, if requested. To request such a scanner, specify

%option reentrant

in your scanner definition. By default, this changes the prototype of yylex to:

int yylex(yyscan_t yyscanner);

where yyscan_t is an opaque pointer. (This is C, so that means it's a void* .) You can read about the details in the Flex manual ; the most important takeaway is that you can ask flex to also generate a header file (with %option header-file ), so that other translation units can refer to the various functions for creating, destroying and manipulating a yyscan_t , and that you need to minimally create one so that yylex has somewhere to store its state. (Ideally, you would also destroy it.) [Note 1].

The expected way to use a reentrant scanner from bison is to enable %option bison-bridge (and %option bison-location if your lexer generates source location information for each token). This will add an additional parameter to the yylex prototype:

int yylex(YYSTYPE *yylval_param, yyscan_t scanner);

With `%option bison-locations', two parameters are added:

int yylex(YYSTYPE *yylval_param,
          YYLTYPE *yylloc_param,
          yyscan_t scanner);

The semantic type YYSTYPE and the location type YYLTYPE are not declared by the flex-generated code. They must appear in the token.h header you #include into your scanner.

The intention of the bison-bridge parameters is to provide a mechanism to return the semantic value yylval to the caller (ie the parser). Since yylval is effectively the same as the parameter yylval_param [Note 2], it will be a pointer to the actual semantic value, so you need to write (for example) yylval->data = ... in your flex actions.

So that's one way to do it.

A possibly simpler alternative to bison-bridge is just to provide your own yylex prototype, which you can do with the macro YY_DECL . For example, you could do something like this (if YYSTYPE were something simple):

#define YY_DECL std::pair<int, YYSTYPE> yylex(yyscan_t yyscanner)

Then a rule could just return the pair:

[0-9]+ {return std::make_pair(NUM, new NumToken(std::stoi(yytext));}

Obviously, there are many variants on this theme.

---

Notes

1. Unfortunately, the generated header includes quite a lot of unnecessary baggage, including a bunch of macro definitions for the standard "globals" which won't work because in a reentrant scanner these variables can only be used in a flex action.

2. The scanner generated with bison-bridge defines yylval as a macro which refers to a field in the opaque state structure, and stores yylval_param into this field. yyget_lval and yyset_lval functions are provided in order to get or set this field from outside of yylex . I don't know why; it seems somewhere between unnecessary and dangerous, since the state will contain the pointer to the value, as supplied in the call to yylex , which may well be a dangling pointer once the call returns.