理解词法分析器中的双缓冲

Question

I'm reading the "Purple Dragon Book" on Compilers as part of my Compiler Construction course at university. I'm having trouble understanding some things about double buffering while scanning input as part of Lexical Analysis.

Here's the text in book:

"

Two pointers to the input are maintained:

1. Pointer lexemeBegin, marks the beginning of the current lexeme, whose extent we are attempting to determine.
2. Pointer forward scans ahead until a pattern match is found; the exact strategy whereby this determination is made will be covered in the balance of this chapter.

"

So, correct me if I'm wrong: One buffer is read, and when the input of that buffer is exhausted, the other buffer is filled with new data from source file, and now buffers are swapped. Forward and beginnig pointers now point to the freshly filled buffer.

My question is, what if some part of lexeme is at the end of current buffer? Then when buffers switch, half of the lexeme will be end of one buffer, and half at the end of new buffer. The pointers will move to new buffer, and we don't exactly know that the other half was left in other buffer?

Sorry if the question is vague, but I've been agonizing for quite some time on how this scenario will be handled. I think same problem will occour using single buffer.

Answer 1

The way Flex handles end of buffer is to relocate the text scanned since lexemeBegin to the start of the buffer, and then fill the buffer from the relocated value of forward to the end of the buffer. So there's only one buffer unless the buffer has to be expanded because the incomplete token completely fills it (in which case there is a brief time when there is a short buffer being copied from and a long buffer being copied to.)

That's not the only way to do it, but it seems to work pretty well in practice.

The original lex used a different strategy; it used a token buffer and read input one character at a time with fgetc until the end of the token was discovered. Each read character was added to the token buffer as it was read. I'm pretty sure it backed up if necessary by copying to the beginning of the token buffer and then rescanning to get the new state, but I'm just working from memory.

The advantage of the flex architecture is that it reads a bit faster, since it's not working a character at a rime, and it can handle big tokens, albeit inefficiently. Lex couldn't handle big tokens because the token buffer ( yytext ) was a static array, not dynamically allocated, and therefore couldn't be grown.

I don't know of a scanner generator which double buffers, but I'll take a look at the Dragon Book in the morning.