Percona Xtradb Cluster failing

Question

I have setup Percona Xtradb cluster with 3 nodes. The first node starts fine with bootstrap, but when I try to start the second node to join the cluster, I get the following error:

2015-08-27 18:08:08 25990 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (a6b3fced-4ca1-11e5-b5da-d69fa186273c): 1 (Operation not permitted)
at galera/src/replicator_str.cpp:prepare_for_IST():463. IST will be unavailable.
2015-08-27 18:08:08 25990 [Note] WSREP: Member 0.0 (db-gc-pxc2) requested state transfer from ' any '. Selected 1.0 (db-gc-pxc1)(SYNCED) as donor.
2015-08-27 18:08:08 25990 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 0)
2015-08-27 18:08:08 25990 [Note] WSREP: Requesting state transfer: success, donor: 1
2015-08-27 18:08:08 25990 [Warning] WSREP: 1.0 (db-gc-pxc1): State transfer to 0.0 (db-gc-pxc2) failed: -12 (Cannot allocate memory)
2015-08-27 18:08:08 25990 [ERROR] WSREP: gcs/src/gcs_group.cpp:int gcs_group_handle_join_msg(gcs_group_t*, const gcs_recv_msg_t*)():731: Will never receive state. Need to abort.
2015-08-27 18:08:08 25990 [Note] WSREP: gcomm: terminating thread
2015-08-27 18:08:08 25990 [Note] WSREP: gcomm: joining thread
2015-08-27 18:08:08 25990 [Note] WSREP: gcomm: closing backend

Below is my cluster config in my.cnf file:

# Galera COnfig
wsrep_cluster_name = pxc
wsrep_cluster_address = gcomm://192.168.2.100,192.168.2.101,10.168.1.102
wsrep_node_address = 10.1.0.101
wsrep_provider = /usr/lib/libgalera_smm.so
wsrep_provider_options = "gcache.size=4G"
wsrep_slave_threads = 32
wsrep_sst_auth = "user:userpass"
wsrep_node_name = node2
#wsrep_sst_method = xtrabackup_throttle
wsrep_sst_method = xtrabackup-v2

What would be causing this error?

FYI, I do have the user and password for wsrep_sst_auth created in the database.

Here is the remainder of the my.cnf it it helps:

back_log = 65535
binlog_format = ROW
character_set_server = utf8
collation_server = utf8_general_ci
datadir = /var/lib/mysql
#default_storage_engine = InnoDB
expand_fast_index_creation = 1
expire_logs_days = 7
innodb_autoinc_lock_mode = 2
innodb_buffer_pool_instances = 6
innodb_buffer_pool_populate = 1
innodb_buffer_pool_size = 6G   # XXX 64GB RAM, 80%
#innodb_data_file_path = ibdata1:64M;ibdata2:64M:autoextend
innodb_file_format = Barracuda
innodb_file_per_table
innodb_flush_log_at_trx_commit = 2
innodb_flush_method = O_DIRECT
innodb_io_capacity = 1600
innodb_large_prefix
innodb_locks_unsafe_for_binlog = 1
#innodb_log_file_size = 64M
innodb_print_all_deadlocks = 1
innodb_read_io_threads = 64
innodb_stats_on_metadata = FALSE
innodb_support_xa = FALSE
innodb_write_io_threads = 64
log-bin = mysqld-bin
log-queries-not-using-indexes
log-slave-updates
long_query_time = 1
max_allowed_packet = 64M
max_connect_errors = 4294967295
max_connections = 4096
min_examined_row_limit = 1000
performance-schema-instrument='%=ON'
port = 3306
relay-log-recovery = TRUE
skip-name-resolve
slow_query_log = 1
slow_query_log_timestamp_always = 1
table_open_cache = 4096
thread_cache = 1024
tmpdir = /srv/tmp
transaction_isolation = REPEATABLE-READ
updatable_views_with_limit = 0
user = mysql
wait_timeout = 60

Answer 1

This would seem to be the root cause:

2015-08-27 18:08:08 25990 [Warning] WSREP: 1.0 (db-gc-pxc1): State transfer to 0.0 (db-gc-pxc2) failed: -12 (Cannot allocate memory)

The new node tries to join the cluster. The new node has no state currently (local UUID is zeroes), and so an IST is not available - this means it needs to run a full SST form the donor node.

Node pxc2 is the joiner and pxc1 is the selected donor; however we get an error from pxc1 that the state transfer failed, which causes the joining to fail.

You should check the logs on the donor node (pxc1) for more detail; but the log we have indicates that it has insufficient memory to run the export of the database. Not knowing your hardware configuration, I can't give a definite response, but most likely your my.cnf is configured to be too memory hungry for your available memory and so the xtrabackup process cannot run, or else the database is too large. Add more memory to the node, or else reduce the allocations in the my.cnf.