[英]Ruby StringScanner used for lexing : how to get the line number?
I am using StringScanner for lexical analysis like this : 我正在使用StringScanner进行词法分析,如下所示:
def next
@scanner.skip(/\s+/)
value,kind=nil,nil
TOKEN_DEF.each{|tok,regex| (kind=tok;break) if @scanner.scan(regex)}
return Token.new(kind,value,@line,@scanner.pos)
end
At first approximation, this works well, except that I can't figure out how to now get the @line number. 初看起来,这很有效,除了我无法弄清楚如何获得@line数。
I have read the doc, where begin_of_line? 我已经阅读了doc,其中begin_of_line? method seems appropriate, but I cannot figure how to use it.
方法似乎合适,但我无法想象如何使用它。
Keep the text that you are scanning in a variable and use 'count' 将您正在扫描的文本保留在变量中并使用“计数”
I use the following in my code: 我在我的代码中使用以下内容:
def current_line_number; @text[0..@scanner.pos].count("\n") + 1; end
This code doesn't seem ready to go and for sure somewhere else more elegant solution, it just should give you something to think about. 这段代码似乎没有准备好去确定其他地方更优雅的解决方案,它应该给你一些思考的东西。
class Retry < StandardError
end
class TextScanner
def initialize(filename)
@lines = IO.readlines(filename)
@fiber = Fiber.new do
@lines.each_with_index do |line, index|
@scanner = StringScanner.new(line)
@scanner.skip(/\s+/)
value, kind = nil, nil
begin
got_token = false
TOKEN_DEF.each do |tok, regex|
if @scanner.scan(regex)
Fiber.yield Token.new(tok, value, index, @scanner.pos)
got_token = true
end
end
raise Retry if got_token
rescue Retry
retry
end
end
"fiber is finished"
end
end
def next
@fiber.resume
end
end
text_scanner = TextScanner('sometextfile')
puts text_scanner.next #=> first token
puts text_scanner.next #=> second token
puts text_scanner.next #=> third token
...
puts text_scanner.next #=> "fiber is finished"
I think I have a simple solution. 我想我有一个简单的解决方案。 Here it is :
这里是 :
def next
@line+=1 while @scanner.skip(/\n/)
@line+=1 if @scanner.bol?
@scanner.skip(/\s+/)
@line+=1 if @scanner.bol?
@scanner.skip(/\s+/)
return :eof if @scanner.eos?
TOKEN_DEF.each { |tok,syntax| (kind=tok;break) if @scanner.scan(syntax)}
return Token.new(kind,nil,@line,@scanner.pos)
end
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.