I am working on seting up two new DB servers in a Master to Slave replication scenario. They are both running Redhat 7.1 and MariaDB 10.1.9. Initially I setup old-style binlog filename/offset position type replication. After some troubleshooting and research I have that working great. When I switch that to use GTID for replication the replication stops working and I get the following erros in the messages log repeating every 60 seconds on reconnect.
Dec 2 01:00:35 ironhide mysqld: 2015-12-02 1:00:35 140155415439104 [ERROR] Error reading packet from server: Table 'mysql.gtid_slave_pos' doesn't exist ( server_errno=1146)
Dec 2 01:01:35 ironhide mysqld: 2015-12-02 1:01:35 140155415439104 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'FIRST' at position 4; GTID position '0-1-47'
Dec 2 01:01:35 ironhide mysqld: 2015-12-02 1:01:35 140155415439104 [ERROR] Error reading packet from server: Table 'mysql.gtid_slave_pos' doesn't exist ( server_errno=1146)
Dec 2 01:02:35 ironhide mysqld: 2015-12-02 1:02:35 140155415439104 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'FIRST' at position 4; GTID position '0-1-47'
I also see this right when mariadb is restarted
[Note] /usr/sbin/mysqld (mysqld 10.1.9-MariaDB) starting as process 31533 ...
Dec 1 22:23:15 ironhide mysqld: 2015-12-01 22:23:15 139849677047936 [Warning] You need to use --log-bin to make --binlog-format work.
Dec 1 22:23:15 ironhide mysqld: 2015-12-01 22:23:15 139849677047936 [Note] InnoDB: Using mutexes to ref count buffer pool pages
Dec 1 22:23:15 ironhide mysqld: 2015-12-01 22:23:15 139849677047936 [Note] InnoDB: The InnoDB memory heap is disabled
Dec 1 22:23:15 ironhide mysqld: 2015-12-01 22:23:15 139849677047936 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
Dec 1 22:23:15 ironhide mysqld: 2015-12-01 22:23:15 139849677047936 [Note] InnoDB: Memory barrier is not used
Dec 1 22:23:15 ironhide mysqld: 2015-12-01 22:23:15 139849677047936 [Note] InnoDB: Compressed tables use zlib 1.2.7
Dec 1 22:23:15 ironhide mysqld: 2015-12-01 22:23:15 139849677047936 [Note] InnoDB: Using Linux native AIO
Dec 1 22:23:15 ironhide mysqld: 2015-12-01 22:23:15 139849677047936 [Note] InnoDB: Not using CPU crc32 instructions
Dec 1 22:23:15 ironhide mysqld: 2015-12-01 22:23:15 139849677047936 [Note] InnoDB: Initializing buffer pool, size = 128.0M
Dec 1 22:23:15 ironhide mysqld: 2015-12-01 22:23:15 139849677047936 [Note] InnoDB: Completed initialization of buffer pool
Dec 1 22:23:15 ironhide mysqld: 2015-12-01 22:23:15 139849677047936 [Note] InnoDB: Highest supported file format is Barracuda.
Dec 1 22:23:15 ironhide mysqld: 2015-12-01 22:23:15 139849677047936 [Note] InnoDB: 128 rollback segment(s) are active.
Dec 1 22:23:15 ironhide mysqld: 2015-12-01 22:23:15 139849677047936 [Note] InnoDB: Waiting for purge to start
Dec 1 22:23:16 ironhide mysqld: 2015-12-01 22:23:16 139849677047936 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.26-74.0 started; log sequence number 1797673
Dec 1 22:23:16 ironhide mysqld: 2015-12-01 22:23:16 139848937633536 [Note] InnoDB: Dumping buffer pool(s) not yet started
Dec 1 22:23:16 ironhide mysqld: 2015-12-01 22:23:16 139849677047936 [Note] Plugin 'FEEDBACK' is disabled.
Dec 1 22:23:16 ironhide mysqld: 2015-12-01 22:23:16 139849677047936 [Note] Server socket created on IP: '0.0.0.0'.
Dec 1 22:23:16 ironhide mysqld: 2015-12-01 22:23:16 139849677047936 [Warning] Neither --relay-log nor --relay-log-index were used; so replication may break when this MySQL server acts as a slave and has his hostname changed!! Please use '--log-basename=#' or '--relay-log=ironhide-relay-bin' to avoid this problem.
Dec 1 22:23:16 ironhide mysqld: 2015-12-01 22:23:16 139849677047936 [Note] /usr/sbin/mysqld: ready for connections.
Dec 1 22:23:16 ironhide mysqld: Version: '10.1.9-MariaDB' socket: '/var/lib/mysql/mysql.sock' port: 3306 MariaDB Server
I was under the impression for a while that the issue was the log file name that it complains about. But I verified that that file exists and is owned by the mysql user,which is what mysql runs under.
My my.cnf file settings look like this.
[mysqld]
server-id=2
bind-address=0.0.0.0
binlog_format=row read_only
log-error = /var/log/mysql/mysql.error.log
#gtid_strict_mode=1
I have tried with the gtid_strict_mode enabled and disabled. It has no affect. I have searched high and low for some posts about this issue. I cannot find them. I do have SElinux enabled and enforcing in the environment. I have looked through the logs for signs of that as well. Any help is greatly appreciated.