Rails 2 database model returns bad encoded string with question marks

Question

Hi I have an oracle database that stores some data. It contains some non english text as well, eg “TEST”. The quote is not the english quote ". The problem is when I retrieve it from Rails 2.2.2 (Ruby 1.8.7), this database model's field value returns question marks in the erg views, so “TEST” becomes ?TEST?. However under Rails 3, it is showing correctly.

The code in the erb that displays the value is

User.first.description

I do set the encoding in the database.yml with the following but does not help

encoding: UTF8 collation: utf8_unicode_ci

Could it because Ruby 1.9 handles the encoding better than Ruby 1.8. Is there a way to fix this problem?

Answer 1

Yes, ruby 1.9 handles encoding different than 1.8. Also, Rails 3 makes encoding easier by trying to make sure everything is in UTF-8.

Most likely your problem is that the string was encoded with Latin-1 and Rails 2 tries to read it as a UTF-8 encoding. There are several monkey patches online that you can try for your database, or you can do a one time script to re-encode all of the fields in your database.

I recommend reading this, for further understanding on how encoding works with Ruby: Encodings, Unabridged (by Yehuda Katz)