无需手动行走/检查即可制作词法分析器

Question

I'm making my own programming language and I'm on the lexer right now. My current approach is to manually walk through the code and check for valid keywords, then append a Token object to a tokens array. But it leaves me with a massive if/else statement that's not only ugly but slow too. I'm struggling to find any resources about this online, and I'm trying to find out if there's a better way to do this - Some regex pattern or something?

Here's the code

class Token:
  def __init__(self, type, value):
    self.type = type
    self.value = value

  def __str__(self):
    return f'Token({self.type}, {self.value})'

  def __repr__(self):
    return self.__str__()


def lex(code):
  tokens = []

  for index in range(len(code)):
    pass # This is where the if/else statement goes

  return tokens

I don't want to use lex or anything. Thanks in advance for the help.

Answer 1

Parser generators can help you get started quickly by helping you define syntax trees and giving you a declarative syntax to describe the lexing & parsing steps.

that's not only ugly but slow too

This seems odd to me. Hand-rolled lexers usually are pretty performant. As long as your syntax doesn't require too much lookahead or back-tracking.

Parser generators typically work based on automata; they build state tables so most of the work is just a loop that at each steps looks up into those tables.

One trick that high-performance, hand-rolled lexers often do is to have a lookup-table that classifies each ASCII character. So the lexing loop looks like

while position < limit:
  code_point = read_codepoint(position)
  if code_point <= MAX_ASCII:
    # switch on CLASSIFICATION[code_point]
  else:
    # Do something else probably identifier related

where CLASSIFICATION stores info that lets you recognize that quote characters inevitably lead to parsing as a quoted string or character literal and space characters can be skipped over and 0-9 inevitably lead to parsing a numeric token.

Some regex pattern or something?

This can work if your lexical grammar is regular .

That probably isn't true if your syntax requires nesting tokens.

For example, JS has non-regularity because template strings can embed expressions:

`string stuff ${ expressionStuff } more string stuff`

so a JS lexer needs to keep state so it knows when a } transitions back into a string state or not.