Ruby StringScanner used for lexing : how to get the line number?

Question

I am using StringScanner for lexical analysis like this :

def next
 @scanner.skip(/\s+/)
 value,kind=nil,nil
 TOKEN_DEF.each{|tok,regex| (kind=tok;break) if @scanner.scan(regex)}
 return Token.new(kind,value,@line,@scanner.pos)
end

At first approximation, this works well, except that I can't figure out how to now get the @line number.

I have read the doc, where begin_of_line? method seems appropriate, but I cannot figure how to use it.

Answer 1

Keep the text that you are scanning in a variable and use 'count'

I use the following in my code:

def current_line_number; @text[0..@scanner.pos].count("\n") + 1; end

Answer 2

This code doesn't seem ready to go and for sure somewhere else more elegant solution, it just should give you something to think about.

class Retry < StandardError
end

class TextScanner
  def initialize(filename)
    @lines = IO.readlines(filename)
    @fiber = Fiber.new do
      @lines.each_with_index do |line, index|
        @scanner = StringScanner.new(line)
        @scanner.skip(/\s+/)
        value, kind = nil, nil
        begin 
          got_token = false
          TOKEN_DEF.each do |tok, regex|
            if @scanner.scan(regex)
              Fiber.yield Token.new(tok, value, index, @scanner.pos)
              got_token = true
            end
          end
          raise Retry if got_token
        rescue Retry
          retry
        end
      end
      "fiber is finished"
    end
  end

  def next
    @fiber.resume
  end
end

text_scanner = TextScanner('sometextfile')
puts text_scanner.next #=> first token
puts text_scanner.next #=> second token
puts text_scanner.next #=> third token
...
puts text_scanner.next #=> "fiber is finished"

Answer 3

I think I have a simple solution. Here it is :

def next
 @line+=1 while @scanner.skip(/\n/)
 @line+=1 if @scanner.bol?
 @scanner.skip(/\s+/)
 @line+=1 if @scanner.bol?
 @scanner.skip(/\s+/)
 return :eof if @scanner.eos?
 TOKEN_DEF.each { |tok,syntax| (kind=tok;break) if @scanner.scan(syntax)}
 return Token.new(kind,nil,@line,@scanner.pos)
end