I am struggling trying to understand why MySQL 5.7.11 semi-sync replication using RBR takes around 6 seconds between 2 VMs on the same physical host. I have verified the network bandwidth b/w the 2 using iperf (around 14 GBits/s).
The latency of 6 seconds is for EVERYTHING including simple statements such as create table, and also insert 1 row into a table.
From the error logs, I see the slave loses and re-establishes connection. The master reports a magic number error, and there is a gap of 5 seconds reported there. Any ideas will be very much appreciated!
-- FROM THE MASTER's LOG
2016-04-09T03:43:06.589976Z 0 [ERROR] Read semi-sync reply magic number error
2016-04-09T03:43:11.589080Z 0 [ERROR] /usr/sbin/mysqld: Got an error reading communication packets
2016-04-09T03:43:11.665142Z 14 [Note] While initializing dump thread for slave with UUID <a25fd780-fdfb-11e5-a290-00505690bd8c>, found a zombie dump thread with the same UUID. Master is killing the zombie dump thread.
2016-04-09T03:43:11.665244Z 13 [Note] Stop semi-sync binlog_dump to slave (server_id: 200)
2016-04-09T03:43:11.665395Z 14 [Note] Start binlog_dump to master_thread_id(14) slave_server(200), pos(, 4)
2016-04-09T03:43:12.591043Z 14 [Note] Start semi-sync binlog_dump to slave (server_id: 200), pos(, 4)
2016-04-09T03:45:12.691794Z 10 [Note] Aborted connection 10 to db: '*****' user: 'root' host: 'localhost' (Got timeout reading communication packets)
-- FROM THE SLAVE's LOG
2016-04-09T03:43:12.634597Z 7 [ERROR] Error reading packet from server for channel '': Lost connection to MySQL server during query (server_errno=2013)
2016-04-09T03:43:12.634683Z 7 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'binlog-3306.000001' at position 3483 for channel ''
2016-04-09T03:45:16.374037Z 6 [Note] Aborted connection 6 to db: '*****' user: 'root' host: 'localhost' (Got timeout reading communication packets)