如何在ruby中使用File.foreach使用UTF-8 BOM读取文件

Question

I would like to read a text file with a an UTF-8 Bom using ruby's file.foreach. The bom is inserted in the file by adding as first line: myFile.write "\" . Download the myFile.txt from this link:

https://wetransfer.com/downloads/b42363faaff561e7e3ca2edbe850d88d20190807164816/c6e4e1

I tried simply to read the file like this

  File.foreach(myFile).with_index do |line, line_num|
    puts "line = " + line
    puts "line.bytes = " + line.bytes.to_s()
    puts "line.bytes.map(&:chr) = " + line.bytes.map(&:chr).to_s()
  end

Problem is that line looks like empty every line of the file. However I can see there is something there using bytes. Also tried using encoding argument of File.foreach as follows

  File.foreach(myFile, :encoding=> 'r:bom|utf-8').with_index do |line,line_num|
    puts "line = " + line
    puts "line.bytes = " + line.bytes.to_s()
    puts "line.bytes.map(&:chr) = " + line.bytes.map(&:chr).to_s()
  end

But I am getting the same results. In both cases seems like ruby recognises the utf-8 bom because puts line.encoding results in 'utf-8'. But I cannot access the chars of line string as usual. For example using myFile.txt the condition below is never triggered.

   if line[0,5] == 'Hello'
     puts "Hello catched"
   end  

Do you know how can I read my files using file.foreach command?

Regards

Answer 1

The problem seems to be that you have a file with mixed encodings of which we can't be sure of the source. So determining how to "read" this data is not as easy as just convert it to whatever. However, you might try this to see what's going on.

File.read('myFile.txt').encode("Windows-1252", invalid: :replace, undef: :replace)
=> "?Hello\nI may contain UTF-8 characters as D\xF8RBLAD\n"

This may not be a complete answer but you may want to refer to this article which covers ideas about how to go about solving your issue.