Bison failed to process token from Flex

Question

When I try to use Bison to process tokens, I find that it works well when the input file is relatively small. However, when the file extends to more than 10 thousands lines, the flex works well(I can get the correct text from yytext, but not sure it is correctly send to Bison), the Bison failed with no rules in process(it succeeded when I made the input file small). I'm not sure if there are some features with Bison like limited buffer size or something like that?

//flex
"setb"|"SETB"                     {return SETB;}

[r][0-9]+                  {yylval.reg = strdup(yytext); return R;}


//bison 

a : R {$$ = e_aR_init($1);}
  | V { $$ = e_aV_init($1);}
  ;

real_instr : SETB a COMMA e COMMA e {$$ = real_setb_init($2, $4, $6);}

yylval.reg has the right string, but when trying to print it from bison by add printf to R's rule, it can't even reach it, making it impossible to match the SETB rule of real_instr.

//================ I discover that yyparse() returns YYNOMEM which indicates the memory exhaustion, any solutions to fix it?

Answer 1

If yyparse is return YYNOMEM , it could be because you're leaking memory by never calling free() on all the tokens whose strings you copy during the lexical scan. You should fix that, in any case.

But it's more likely that it is caused by overflowing the parser stack, which normally happens because you chose to use a right-recursive grammar for a repeated element. Bottom-up parsers, such as the ones generated by Bison, do not need unlimited parse stack in order to parse left-recursive grammars, which is why it is always recommended that productions be left-recursive unless you have some other requirement.

With top-down parsers, you cannot use left-recursion. But that doesn't mean that left recursion is bad. It only means that you can't use it in a top-down parser. In other words, it's a feature of top-down parsing, not of left-recursion. Top-down parsers also chew through stack space with right-recursive productions, although some parser generators will produce while loops in order to avoid recursion, which avoids stack usage in some cases.