MySQL collation issue with Entity Framework Core

Question

My asp.net core web application is hitting is MySQL database. I generate migration and update database, then I see the collation of table which is different on different machines. On my machine its latin1 - default collation and on a different machine, it is utf8 - default collation .

So due to this behavior, I'm getting an exception while inserting Japanese or some other language work like 日本語, Español México or 简体中文 into a table which has a varchar(64) column. It works fine on another machine which has utf8 - default collation on this table.

EF Core Version: 2.0.1

MySql: 5.7.22**

Is this EF Core issue? Or is there any workaround to generate the database with the proper collation or encoding?

Answer 1

It is not EF Core issue, if you have default collation configured in your MySql then that will be picked by MySql. So collation is database system configuration thing. Probably you have to change your default collation on your machine.

Other option is to add after creating table statement in migration:

Sql('alter table <some_table> convert to character set utf8 collate utf8_unicode_ci');

Answer 2

Configure every column manually to a specific CharSet and Collation, In case you cannot change server/schema default CharSet or Collation (you are not the Server administrator)

using System;
using System.Collections.Generic;
using System.Linq;
using System.Threading.Tasks;
using Microsoft.EntityFrameworkCore;
using Microsoft.EntityFrameworkCore.Metadata;
using Microsoft.EntityFrameworkCore.Metadata.Builders;
using Pomelo.EntityFrameworkCore.MySql.Extensions;
using Pomelo.EntityFrameworkCore.MySql.Infrastructure;

namespace Data
{
    public static class Extensions
    {
        internal static PropertyBuilder<string>  TextField(this PropertyBuilder<string> string_column, int max_length)
        {
            string_column.TextField(CharSet.Utf8Mb4, "utf8mb4_unicode_ci", max_length);
            return string_column;
        }
        internal static PropertyBuilder<string>  TextField(this PropertyBuilder<string> string_column, CharSet char_set, string collation, int max_length)
        {
            string_column.HasColumnType($"VARCHAR({max_length}) CHARACTER SET {char_set} COLLATE {collation}");
            return string_column;
        }
    }
}

Set character set of each property

        protected override void OnModelCreating(ModelBuilder builder)
        {
            ...
            builder.Entity<TABLE>().Property(t => t.Name).TextField(1024);
            ...
        }

Check the result by query column collation settings

> mysql.exe --login-path=mysql1 --database=test1 -e "select column_name, character_set_name, collation_name from information_schema.columns where table_name = 'TABLE';"

+-------------+--------------------+--------------------+
| column_name | character_set_name | collation_name     |
+-------------+--------------------+--------------------+
| Id          | NULL               | NULL               |
| Name        | utf8mb4            | utf8mb4_unicode_ci |
| CreatedTime | NULL               | NULL               |
| UpdatedTime | NULL               | NULL               |
+-------------+--------------------+--------------------+

My Entity Framework Project lib with MySQL 5.7.21 .

 <ItemGroup>
    ...
    <PackageReference Include="Microsoft.EntityFrameworkCore.Tools" Version="5.0.1">
      <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
      <PrivateAssets>all</PrivateAssets>
    </PackageReference>
    <PackageReference Include="Pomelo.EntityFrameworkCore.MySql" Version="5.0.0-alpha.2" />
  </ItemGroup>

Noted the following can set CharSet but Collation is fixed to utf8mb4_general_ci .

            services.AddPooledDbContextFactory<ApplicationDbContext>(
                options => options
                    .UseMySql(
                        Configuration["ConnectionString"],
                        new MySqlServerVersion(new Version(5, 7, 21)),
                        mySqlOptions =>
                        {                          
                            mySqlOptions
                               .CharSetBehavior(CharSetBehavior.AppendToAllColumns);
                            mySqlOptions.CharSet(CharSet.Utf8Mb4);
                        })
                    .EnableSensitiveDataLogging()
                    .EnableDetailedErrors()
            );