MySQL Replication stuck

Question

I set up MySQL Replication from a Master to a Slave. "SHOW SLAVE STATUS\\G" is showing a continually increasing Seconds_Behind_Master. Here's three consecutive results from oldest (top) to newest (bottom). Anybody know what's wrong?............................................................................................................................................................................................................................................................

1

    mysql> SHOW SLAVE STATUS\G
*************************** 1. row ***************************
               Slave_IO_State: Queueing master event to the relay log
                  Master_Host: <excluding>
                  Master_User: <excluding>
                  Master_Port: 3306
                Connect_Retry: 60
              Master_Log_File: mysql-bin.002723
          Read_Master_Log_Pos: 893947806
               Relay_Log_File: design-hotels-database-relay-bin.000002
                Relay_Log_Pos: 7692403
        Relay_Master_Log_File: mysql-bin.002723
             Slave_IO_Running: Yes
            Slave_SQL_Running: Yes
              Replicate_Do_DB:
          Replicate_Ignore_DB: starwood_sop,starwood_blp
           Replicate_Do_Table:
       Replicate_Ignore_Table:
      Replicate_Wild_Do_Table:
  Replicate_Wild_Ignore_Table:
                   Last_Errno: 0
                   Last_Error:
                 Skip_Counter: 0
          Exec_Master_Log_Pos: 192464416
              Relay_Log_Space: 709176017
              Until_Condition: None
               Until_Log_File:
                Until_Log_Pos: 0
           Master_SSL_Allowed: No
           Master_SSL_CA_File:
           Master_SSL_CA_Path:
              Master_SSL_Cert:
            Master_SSL_Cipher:
               Master_SSL_Key:
        Seconds_Behind_Master: 76613
Master_SSL_Verify_Server_Cert: No
                Last_IO_Errno: 0
                Last_IO_Error:
               Last_SQL_Errno: 0
               Last_SQL_Error:
  Replicate_Ignore_Server_Ids:
             Master_Server_Id: 3
                  Master_UUID:
             Master_Info_File: /var/lib/mysql/master.info
                    SQL_Delay: 0
          SQL_Remaining_Delay: NULL
      Slave_SQL_Running_State: Reading event from the relay log
           Master_Retry_Count: 86400
                  Master_Bind:
      Last_IO_Error_Timestamp:
     Last_SQL_Error_Timestamp:
               Master_SSL_Crl:
           Master_SSL_Crlpath:
           Retrieved_Gtid_Set:
            Executed_Gtid_Set:
                Auto_Position: 0
         Replicate_Rewrite_DB:
                 Channel_Name:
           Master_TLS_Version:
1 row in set (0.04 sec)

2

mysql> SHOW SLAVE STATUS\G
*************************** 1. row ***************************
               Slave_IO_State: Queueing master event to the relay log
                  Master_Host: <excluding>
                  Master_User: <excluding>
                  Master_Port: 3306
                Connect_Retry: 60
              Master_Log_File: mysql-bin.002723
          Read_Master_Log_Pos: 908310158
               Relay_Log_File: design-hotels-database-relay-bin.000002
                Relay_Log_Pos: 7906291
        Relay_Master_Log_File: mysql-bin.002723
             Slave_IO_Running: Yes
            Slave_SQL_Running: Yes
              Replicate_Do_DB:
          Replicate_Ignore_DB: starwood_sop,starwood_blp
           Replicate_Do_Table:
       Replicate_Ignore_Table:
      Replicate_Wild_Do_Table:
  Replicate_Wild_Ignore_Table:
                   Last_Errno: 0
                   Last_Error:
                 Skip_Counter: 0
          Exec_Master_Log_Pos: 192678304
              Relay_Log_Space: 723538369
              Until_Condition: None
               Until_Log_File:
                Until_Log_Pos: 0
           Master_SSL_Allowed: No
           Master_SSL_CA_File:
           Master_SSL_CA_Path:
              Master_SSL_Cert:
            Master_SSL_Cipher:
               Master_SSL_Key:
        Seconds_Behind_Master: 76698
Master_SSL_Verify_Server_Cert: No
                Last_IO_Errno: 0
                Last_IO_Error:
               Last_SQL_Errno: 0
               Last_SQL_Error:
  Replicate_Ignore_Server_Ids:
             Master_Server_Id: 3
                  Master_UUID:
             Master_Info_File: /var/lib/mysql/master.info
                    SQL_Delay: 0
          SQL_Remaining_Delay: NULL
      Slave_SQL_Running_State: Reading event from the relay log
           Master_Retry_Count: 86400
                  Master_Bind:
      Last_IO_Error_Timestamp:
     Last_SQL_Error_Timestamp:
               Master_SSL_Crl:
           Master_SSL_Crlpath:
           Retrieved_Gtid_Set:
            Executed_Gtid_Set:
                Auto_Position: 0
         Replicate_Rewrite_DB:
                 Channel_Name:
           Master_TLS_Version:
1 row in set (0.06 sec)

3

mysql> SHOW SLAVE STATUS\G
*************************** 1. row ***************************
               Slave_IO_State: Queueing master event to the relay log
                  Master_Host: <excluding>
                  Master_User: <excluding>
                  Master_Port: 3306
                Connect_Retry: 60
              Master_Log_File: mysql-bin.002723
          Read_Master_Log_Pos: 945765663
               Relay_Log_File: design-hotels-database-relay-bin.000002
                Relay_Log_Pos: 8389913
        Relay_Master_Log_File: mysql-bin.002723
             Slave_IO_Running: Yes
            Slave_SQL_Running: Yes
              Replicate_Do_DB:
          Replicate_Ignore_DB: starwood_sop,starwood_blp
           Replicate_Do_Table:
       Replicate_Ignore_Table:
      Replicate_Wild_Do_Table:
  Replicate_Wild_Ignore_Table:
                   Last_Errno: 0
                   Last_Error:
                 Skip_Counter: 0
          Exec_Master_Log_Pos: 193161926
              Relay_Log_Space: 760993874
              Until_Condition: None
               Until_Log_File:
                Until_Log_Pos: 0
           Master_SSL_Allowed: No
           Master_SSL_CA_File:
           Master_SSL_CA_Path:
              Master_SSL_Cert:
            Master_SSL_Cipher:
               Master_SSL_Key:
        Seconds_Behind_Master: 76870
Master_SSL_Verify_Server_Cert: No
                Last_IO_Errno: 0
                Last_IO_Error:
               Last_SQL_Errno: 0
               Last_SQL_Error:
  Replicate_Ignore_Server_Ids:
             Master_Server_Id: 3
                  Master_UUID:
             Master_Info_File: /var/lib/mysql/master.info
                    SQL_Delay: 0
          SQL_Remaining_Delay: NULL
      Slave_SQL_Running_State: Reading event from the relay log
           Master_Retry_Count: 86400
                  Master_Bind:
      Last_IO_Error_Timestamp:
     Last_SQL_Error_Timestamp:
               Master_SSL_Crl:
           Master_SSL_Crlpath:
           Retrieved_Gtid_Set:
            Executed_Gtid_Set:
                Auto_Position: 0
         Replicate_Rewrite_DB:
                 Channel_Name:
           Master_TLS_Version:
1 row in set (0.04 sec)

Answer 1

Well, the Exec_Master_Log_Pos is also increasing, which indicates the SQL thread is actually working and applying changes. Just not fast enough to keep up with the incoming logs, indicated by the rate of increase of Read_Master_Log_Pos

    Exec-pos              Read-pos
1:  192464416             893947806
2:  192678304 (+213,888)  908310158 (+14,362,352
3:  193161926 (+483,622)  945765663 (+37,455,505) 

So the incoming logs (Read-pos) are arriving at a many times higher rate than the rate at which replication is able to execute the changes on the slave instance.

Replication lag is a complex problem, and there are many causes of it. There are lots of configuration settings you can use to help improve throughput on the slave. You didn't describe anything about your current configuration or the nature of your database changes or the tables that are changing, so I can't make really informed recommendations. But below are some typical changes I usually make:

innodb_buffer_pool_size = <high> # depends on server RAM and other factors 
innodb_log_file_size = 1G # or more
innodb_flush_log_at_trx_commit = 2
master_info_repository = TABLE
relay_log_info_repository = TABLE
relay_log_recovery = ON
sync_master_info = 0
binlog_format = ROW # make sure all tables have PRIMARY KEY constraints

Refer to MySQL documentation for details on these options.

Even with all these tuning tricks, eventually you might hit a limit, that is the slave's ability to replay in a single thread the changes that were created in many threads on the master.

You might be able to employ a multi-threaded slave, but this isn't as simple as it sounds, so you need to read more about it before you try this.

Even with tuning and multi-threaded slave, it's possible you have reached the limit of your hardware. It simply can't process the changes fast enough. You could upgrade the hardware to faster CPU and faster storage.

Another option is to reduce the rate of changes to your master database. Either throttle changes on the master, or else deploy multiple masters with their respective slaves, and balance the changes over the multiple masters.