I'm really new to using the percona tools, and I'm trying to learn how to use them correctly.
I’m having a problem with replication drift in my 4 node mysql setup. I’m running a wiki on this setup. And when I create a new page or edit text in the wiki, the changes will only get saved to one of the 4 db nodes and not get replicated to the others,
I'm running MariaDB-server-10.0.21-1 on all 4 database nodes.
I’ll then have to go into the wiki config and point the wiki to one DB at a time until I find the missing content. And then I’ll have to replicate the database by hand to the other 3 nodes.
This will happen even tho all indicators of replication seem fine on all 4 db nodes.
For example this is what I see on all of them as demonstrated on the 1st node:
[root@db1:~] #mysql -e "show slave status \G" | egrep "Slave_IO_Running|Slave_SQL_Running|Seconds_Behind_Master"
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
Seconds_Behind_Master: 0
It’s a real pain! So I’m hoping to solve this problem with some help from the percona toolkit.
I just ran a pt-table-checksum command on my 4 node database cluster, and I got this response back:
[root@db1:~] #pt-table-checksum h=localhost -d jfwiki -t text -u admin -p secret --no-check-binlog-format --no-version-check
Waiting for the --replicate table to replicate to db4...
Waiting for the --replicate table to replicate to db4...
Waiting for the --replicate table to replicate to db4...
Waiting for the --replicate table to replicate to db4...
Waiting for the --replicate table to replicate to db4...
Waiting for the --replicate table to replicate to db4...
Waiting for the --replicate table to replicate to db4...
Waiting for the --replicate table to replicate to db4...
Waiting for the --replicate table to replicate to db4...
Waiting for the --replicate table to replicate to db4...
Waiting for the --replicate table to replicate to db4...
Waiting for the --replicate table to replicate to db4...
And it just keeps repeating that message, and nothing else seems to happen. I even let this command run all night with no change. The same message just keeps repeating with no other output.
What do these messages mean? And how can I get past this so that I can really check the state of replication with this tool?
Here’s the full output of the show slave status command in case that provides some more context.
MariaDB [(none)]> show slave status \G
*************************** 1. row ***************************
Slave_IO_State: Waiting for master to send event
Master_Host: db2.example.com
Master_User: jf_slave
Master_Port: 3306
Connect_Retry: 60
Master_Log_File: mariadb-bin.000082
Read_Master_Log_Pos: 376
Relay_Log_File: db4-relay-bin.000062
Relay_Log_Pos: 537
Relay_Master_Log_File: mariadb-bin.000082
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
Replicate_Do_DB:
Replicate_Ignore_DB:
Replicate_Do_Table:
Replicate_Ignore_Table:
Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table:
Last_Errno: 0
Last_Error:
Skip_Counter: 0
Exec_Master_Log_Pos: 376
Relay_Log_Space: 832
Until_Condition: None
Until_Log_File:
Until_Log_Pos: 0
Master_SSL_Allowed: Yes
Master_SSL_CA_File: /opt/mysql/ca.crt
Master_SSL_CA_Path:
Master_SSL_Cert: /opt/mysql/db4.example.com.crt
Master_SSL_Cipher:
Master_SSL_Key: /opt/mysql/db4.example.com.key
Seconds_Behind_Master: 0
Master_SSL_Verify_Server_Cert: No
Last_IO_Errno: 0
Last_IO_Error:
Last_SQL_Errno: 0
Last_SQL_Error:
Replicate_Ignore_Server_Ids:
Master_Server_Id: 2
Master_SSL_Crl: /opt/mysql/ca.crt
Master_SSL_Crlpath:
Using_Gtid: No
Gtid_IO_Pos:
1 row in set (0.00 sec)
Thanks!