从MySQL导入海量数据到Elasticsearch

Question

I am trying to import 32m rows from MySQL to Elasticsearch using Logstash, it works fine but breaks when reached 3,5m. Checked MySQL, Logstash it works fine, the problem in Elasticsearch please see logs:

[2018-08-14T23:06:44,299][WARN ][o.e.x.s.a.s.m.NativeRoleMappingStore] [4OtmyM2] Failed to clear cache for realms [[]]
[2018-08-14T23:06:44,345][INFO ][o.e.l.LicenseService     ] [4OtmyM2] license [23fbbbff-0ba9-44f5-be52-7f5a6498dbd1] mode [basic] - valid
[2018-08-14T23:06:44,368][INFO ][o.e.g.GatewayService     ] [4OtmyM2] recovered [1] indices into cluster_state
[2018-08-14T23:06:46,120][INFO ][o.e.c.r.a.AllocationService] [4OtmyM2] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[clustername][2]] ...]).
[2018-08-14T23:55:55,780][INFO ][o.e.m.j.JvmGcMonitorService] [4OtmyM2] [gc][2953] overhead, spent [378ms] collecting in the last [1s]

I've increased heap size to 2GB, but it still can't handle it. The configuration file for migration below:

input {
    jdbc {
        jdbc_connection_string => "jdbc:mysql://localhost:3306/clustername?useCursorFetch=true"
        jdbc_user => "USER"
        jdbc_password => "PSWD"
        jdbc_validate_connection => true
        jdbc_driver_library => "/usr/share/java/mysql-connector-java-5.1.42.jar"
        jdbc_driver_class => "com.mysql.jdbc.Driver"
        jdbc_paging_enabled => "true"
        #jdbc_fetch_size => "50000"
        jdbc_page_size => 100000
        statement => "SELECT * FROM `video` ORDER by `id` ASC LIMIT 100000 OFFSET 3552984"
    }
}

Thank you for any advice.

Answer 1

You haven't provided enough data to help diagnose the problem. To properly index large amounts of data, you have to truly understand what the data is and how much storage it's going to take and how much memory it's going to use.

Elasticsearch is not magic. You have to understand some things if you are going beyond a simple proof of concept. When you see things like gc overhead taking a significant time, you have to assume that you haven't properly sized your Elasticsearch cluster.

Things that you need to consider:

• How many shards do I need?
  • The default # in the elasticsearch config file of 5 may work, or it may be too many or too few.
  • Too many shards can cause elasticsearch to run out of memory. Too few shards can cause bad performance.
  • To aid in cluster recovery, your shards should not be large -- somewhere in the 2 GB to 4GB range should be considered "large"
  • Elasticsearch provides APIs to see how many shards you're using and how big they are

• How much memory does elasticsearch need?

  • For a data node, the recommended usage is 50% of the system's RAM
  • The 50% recommendation is related to allowing the OS to use the other 50% for disk cache
  • if you are running other things on the nodes, you probably need to re-architect or adjust if performance allows
  • If your data is time-series based, you should probably be using time-series named indexes (with the frequency being yearly/monthly/weekly/daily depending on how many records per day are generated

• How many nodes do you need

  • Without a 2nd node, you can't have replicas.
  • Without replicas, you will eventually lose data
  • You need to have an odd number of master eligible nodes (otherwise you can get into a split-brain situation where your cluster is partitioned)
  • More nodes are better -- especially if you need a lot of shards
• How big is your data
  • you can reduce the size by configuring fields as keyword only fields (ie if you don't need to search certain fields, or only need to search based on _all)
  • How many fields are you using per record -- more fields = more ram per row

There are many more things that you need to consider, but as a general rule, try to isolate where your fault is at -- ie remove SQL server / logstash from the mix by generating some random amount of data that looks like your real data so that you can gather the metrics needed to properly size your cluster.