Rails migrations suddenly creating tables with different charsets seemingly randomly

Question

We've been using Rails (version 2.3.8, Ruby version 1.8.7, if it's relevant) for some time now and everything has seemed to work fine. All the tables in our MySQL version 5.5.20 DBMS were using the default latin-1 charset, which is I guess fine since we don't have international users (although our database.yml actually specifies utf8... Nobody had noticed the databases weren't actually coming out that way before now).

However I just discovered that for one of our last applications, the tables in that particular database are using utf8. We're now adding on to this application, and on our development machines and on the test server, it seems to be totally random as to which charset newly created tables end up with. On Friday I deployed to our test server and two tables were created at that time, one of which is using latin-1 and the other is using utf8. On my machine they were both utf8. Since we're using foreign keys between these tables, this is a problem.

I've found loads of info on fixing charset issues , migrating from one charset to another , etc. but nothing to explain why we'd suddenly be having this issue, and why it's switching back and forth.

The migrations for the two tables are

create_table :charges, :id => false do |t|
  t.string :id
  t.decimal :amount
  t.decimal :charge
  t.text :description
  t.string :charge_type_ref_id
  t.string :business_id

  t.timestamps
end

and

create_table :charge_type_refs, :id => false do |t|
  t.string :id
  t.string :name
  t.string :code
  t.boolean :active
  t.decimal :fee

  t.timestamps
end

Our database.yml looks like

development:
adapter: mysql
encoding: utf8
reconnect: false
database: database_development
pool: 5
username: username
password: password
socket: /tmp/mysql.sock

What can I do to stop this happening? If I must I'll resort to specifying the charset in each migration, but this seems a bit ridiculous. I'd also just really like to know why this is happening, because it seems very strange. It's just effecting this one database for this one application, and all the rest of our apps and databases are uniform. I don't get it.

Edit: Actually looking closer at all the databases, there are one or two tables in most of them that are utf8. So the problem has been around before; there just weren't foreign keys between those tables so we never noticed before. The database we're currently working on seems effected to a much larger degree than the others though.

---

The solution in the accepted answer, adding

[mysqld]
character-set-server  = utf8
collation-server = utf8_general_ci

to my.cnf worked perfectly on my development machine. I recreated the tables multiple times and it worked every time. For some reason though when I tried this on our test server, it didn't work. I verified that the settings in my.cnf had taken effect by looking at the output of SHOW VARIABLES LIKE 'char%'; and SHOW VARIABLES LIKE 'collation%'; All the variables were set to utf8 or utf8_general_ci as they were supposed to be. Yet somehow some of the tables that were created still came out latin1. I tried this multiple times with the same result.

It seems like this solution will work some of the time, but not always. For whatever bizarre reason. I gave up and added :options => 'DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci' to all of the table creating migrations for our test and development servers.

---

After contacting Oracle for support (I had originally thought this was a Rails issue, which is why I didn't do that in the first place) it turns out that for existing databases, you have to look at the database specific charset/collation settings. Those override the server-level settings when creating new tables. So if anybody else experiences this, first you've got to reset the database to utf8 with ALTER DATABASE databasename CHARACTER SET=utf8; . If you also add the settings in my.cnf then for new databases going forward they'll be created with the correct character set, theoretically.

Answer 1

I suspect that the charset needs to be set on the server side. I had problems like this until I set the charset in the /etc/mysql/my.cnf according to the docs:

http://dev.mysql.com/doc/refman/5.0/en/charset-applications.html

[mysqld] character-set-server = utf8 collation-server = utf8_general_ci

Also good discussion here Change MySQL default character set to UTF-8 in my.cnf? https://dba.stackexchange.com/questions/8288/how-to-configure-global-charset-on-mysql