带有rails CSV的MalformedCSVError（FasterCSV）

Question

我现在正试图在rails中解析一些CSV时遇到严重问题。 基本上我的应用程序让用户上传CSV文件。 然后，应用程序转换文件以确保它是UTF-8格式，然后尝试解析并处理它。 每当应用程序尝试解析它时，我会收到MalformedCSVError，指出“第1行非法引用”

现在我不知道的是，如果我将原始文件复制到一个新文档并保存，那么我可以在rails控制台上解析它而不会出现问题。

如果我尝试解析原始文件，它会抱怨UTF-8编码的字符无效（该文件不是UTF-8，因此应用程序将其转换）

如果我尝试解析应用已转换为UTF-8的文件并将行结尾更改为LF，则无法解析。

如果我在应用程序生成的版本和我制作的复制/粘贴版本（有效）之间进行文件差异，则有0个差异，所以我真的无法弄清楚为什么一个是可解析的，而另一个则不是。

有什么建议么？ 我的应用正在处理文件如下：

def create
@survey = Survey.new(params[:survey])

# Now we need to try and convert this to UTF-8 if it isn't already
 encoded = File.read(@survey.survey_data.current_path)
encoding = CharlockHolmes::EncodingDetector.detect(encoded)

# We've got a guess at the encoding, 
# so we can try and convert it but it 
# may still fail so we need to handle 
# that
begin
  re_encoded = CharlockHolmes::Converter.convert(encoded, encoding[:encoding], 'UTF-8')
  re_encoded = re_encoded.gsub(/\r\n?/, "\n")

  # Now replace the uploaded file
  File.open(@survey.survey_data.current_path, 'w') { |f|
    f.write(re_encoded)
  }
rescue ArgumentError
  puts "UH OH!!!!!"
end

puts "#{@survey.survey_data.current_path}"
@parsed = CSV.read(@survey.survey_data.current_path)

结束

文件上传宝石是CarrierWave，如果这有任何区别。

请有人帮助我，因为这让我疯了！

编辑

错误表明它在第1行。第1行（假设它不是从0索引）是

"Survey","RD","GarrysMDs","NigelsMDs","PaulsMDs","StephensMDs","BrinleyJ","CarolineP","DaveL","GrantR","GregS","Kent","NeilC","NicolaP","AndyC","DarrenS","DeanB","KarenF","PaulR","RichardF","SteveG","BrianG","GordonA","NickD","NickR","NickT","RayL","SimonH","EdmondH","JasonF","MikeS","SamanthaN","TimB","TravisF","AlanS","Q1","Q2","Q3","Q4","Q5","Q6","Q7","Q8PM","Q8N","Q9","Q10","Q11","Q12","Q13","Q14","Q15","Q16PM","Q16N","Q17PM","Q17N","Q18PM","Q18N","Q19","Q20","Q21","Q22","comment","Q23.1","Q23.2","Q23.3","TQ23.1","TQ23.2","VPM","VN","VQ1","VQ2","VQ3","VQ4","VQ5","VQ6","VQ7","VQ8N","VQ8PM","VQ9","VQ10","VQ11","VQ12","VQ13","VQ14","VQ15","VQ16","VQ16N","VQ16PM","VQ17","VQ17N","VQ17PM","VQ18","VQ18N","VQ18PM","VQ19","VQ20","VQ21","VQ22","VQ23.1","VQ23.2","VQ23.3","VRD","XQ16","XQ17","XQ18"

Answer 1

那太刺激了！

原来该文件有一个BOM导致CSV解析器中断。 加载文件

CSV.open("path/to/file.csv", "rb:bom|encoding")

允许它完美地解析它！ 所以很烦恼追踪它需要多长时间，但它现在正在工作，现在也不需要转换为UTF-8！