I have a two separate Galera clusters, will call them C1 and C2. C1 is a main cluster and has a three nodes, C2 is emergency and has only one node. C1 connected with C2 by GTID replication for providing completely identity. The main problem that I can't increase a number of nodes in C2 cluster. I get these errors:
2016-08-02 12:54:00 19067 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2016-08-02 13:00:58 19067 [Warning] WSREP: RBR event 1 Gtid apply warning: 1, 12305593
2016-08-02 13:00:58 19067 [Warning] WSREP: Failed to apply app buffer: seqno: 12305593, status: 1
at galera/src/trx_handle.cpp:apply():351
Retrying 2th time
2016-08-02 13:00:58 19067 [Warning] WSREP: RBR event 1 Gtid apply warning: 1, 12305593
2016-08-02 13:00:58 19067 [Warning] WSREP: Failed to apply app buffer: seqno: 12305593, status: 1
at galera/src/trx_handle.cpp:apply():351
Retrying 3th time
2016-08-02 13:00:58 19067 [Warning] WSREP: RBR event 1 Gtid apply warning: 1, 12305593
2016-08-02 13:00:58 19067 [Warning] WSREP: Failed to apply app buffer: seqno: 12305593, status: 1
at galera/src/trx_handle.cpp:apply():351
Retrying 4th time
2016-08-02 13:00:58 19067 [Warning] WSREP: RBR event 1 Gtid apply warning: 1, 12305593
2016-08-02 13:00:58 19067 [ERROR] WSREP: Failed to apply trx: source: 377c1c2e-5885-11e6-aed8-1e04b1e53bff version: 3 local: 0 state: APPLYING flags: 1 conn_id: 346 trx
_id: 23137618 seqnos (l: 5, g: 12305593, s: 12305592, d: 12305592, ts: 16848639299414398)
2016-08-02 13:00:58 19067 [ERROR] WSREP: Failed to apply trx 12305593 4 times
2016-08-02 13:00:58 19067 [ERROR] WSREP: Node consistency compromized, aborting...
2016-08-02 13:00:58 19067 [Note] WSREP: Closing send monitor...
2016-08-02 13:00:58 19067 [Note] WSREP: Closed send monitor.
2016-08-02 13:00:58 19067 [Note] WSREP: gcomm: terminating thread
2016-08-02 13:00:58 19067 [Note] WSREP: gcomm: joining thread
2016-08-02 13:00:58 19067 [Note] WSREP: gcomm: closing backend
The main settings that relates to the issue: C1 and C2:
binlog_format=ROW
wsrep_sst_method=xtrabackup-v2
server_id = 1 (C1); server_id = 101 (C2)
wsrep_OSU_method = 'TOI'
What is most interesting, that with simple async replication (not GTID) all worked properly. Have you any ideas, guys?