Ruby / Rails CSV解析，UTF-8中的无效字节序列

Question

I am trying to parse a CSV file generated from an Excel spreadsheet.

Here is my code

require 'csv'
file = File.open("input_file")
csv = CSV.parse(file)

But I get this error

ArgumentError: invalid byte sequence in UTF-8

I think the error is because Excel encodes the file into ISO 8859-1 (Latin-1) and not in UTF-8

Can someone help me with a workaround for this issue, please

Thanks in advance.

Answer 1

You need to tell Ruby that the file is in ISO-8859-1. Change your file open line to this:

file=File.open("input_file", "r:ISO-8859-1")

The second argument tells Ruby to open read only with the encoding ISO-8859-1.

Answer 2

Specify the encoding with encoding option:

CSV.foreach(file.path, headers: true, encoding:'iso-8859-1:utf-8') do |row|
  ...
end

Answer 3

You can supply source encoding straight in the file mode parameter:

CSV.foreach( "file.csv", "r:windows-1250" ) do |row|
   <your code>
end

Answer 4

将文件保存为utf-8，除非由于某种原因需要以不同方式保存，在这种情况下，您可以在读取文件时指定编码集

Answer 5

将第二个参数"r:ISO-8859-1"为File.open("input_file","r:ISO-8859-1" )

Answer 6

I had this same problem and was just using google spreadsheets and then downloading as a CSV. That was the easiest solution.

Then I came across this gem

https://github.com/singlebrook/utf8-cleaner

Now I don't need to worry about this issue at all. Hope this helps!

Answer 7

If you have only one (or few) file, so when its not needed to automatically declare encoding on whatever file you get from input, and you have the contents of this file visible in plaintext (txt, csv etc) separated with ie semicolon, you can create new file with .csv extension manually, and paste the contents of your file there, then parse the contents like usual.

Keep in mind, that this is a workaround, but in need of parsing in linux only one big excel file, converted to some flavour of csv, it spares time on experimenting with all those fancy encodings