We have MySQL replication setup and it has been running without issue for some time. This server acts as a slave and a master (it replicates through to another server). This morning I received an alert to say it has stopped:
mysql> SHOW SLAVE STATUS\G;
*************************** 1. row ***************************
Slave_IO_State: Connecting to master
Master_Host: 123.456.78.90
Master_User: replication
Master_Port: 3306
Connect_Retry: 60
Master_Log_File: binary-log.000006
Read_Master_Log_Pos: 225898044
Relay_Log_File: mysqld-relay-bin.000018
Relay_Log_Pos: 4
Relay_Master_Log_File: binary-log.000006
Slave_IO_Running: Connecting
Slave_SQL_Running: Yes
Replicate_Do_DB:
Replicate_Ignore_DB:
Replicate_Do_Table:
Replicate_Ignore_Table:
Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table: mysql.%
Last_Errno: 0
Last_Error:
Skip_Counter: 0
Exec_Master_Log_Pos: 225898044
Relay_Log_Space: 107
Until_Condition: None
Until_Log_File:
Until_Log_Pos: 0
Master_SSL_Allowed: Yes
Master_SSL_CA_File: /var/lib/mysql/ssl/ca-cert.pem
Master_SSL_CA_Path:
Master_SSL_Cert: /var/lib/mysql/ssl/client-cert.pem
Master_SSL_Cipher:
Master_SSL_Key: /var/lib/mysql/ssl/client-key.pem
Seconds_Behind_Master: NULL
Master_SSL_Verify_Server_Cert: No
Last_IO_Errno: 2026
Last_IO_Error: error connecting to master 'replication@134.213.65.178:3306' - retry-time: 60 retries: 86400
Last_SQL_Errno: 0
Last_SQL_Error:
Replicate_Ignore_Server_Ids:
Master_Server_Id: 0
1 row in set (0.00 sec)
ERROR:
No query specified
If I try connect from the same machine:
[user@host~]# mysql -u replication -p -h 123.456.78.90 Enter password: Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 35069260 ...
And permissions from the master:
mysql> SHOW GRANTS;
+-------------------------------------------------------------------------------------------------------------------------------------+
| Grants for replication@123.456.78.90 |
+-------------------------------------------------------------------------------------------------------------------------------------+
| GRANT REPLICATION SLAVE ON *.* TO 'replication'@'123.456.78.90' IDENTIFIED BY PASSWORD '*E2F700F7365XXXXXXXXXX9F3A4C5C17C' |
+-------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.01 sec)
The error log on the slave shows:
150709 20:59:00 [ERROR] Slave I/O: error connecting to master 'replication@123.456.78.90:3306' - retry-time: 60 retries: 86400, Error_code: 2026
It was at that point I remembered last night updates were run:
Updating:
mysql55w x86_64 5.5.44-1.w6
mysql55w-libs x86_64 5.5.44-1.w6
mysql55w-server x86_64 5.5.44-1.w6
But I've not been able to find any related information to this particular update. The error, 2026
, is the following:
Error: 2026 (CR_SSL_CONNECTION_ERROR)
Message: SSL connection error: %s
The certificates haven't changed and the paths are specified correctly on the slave as show at the start of the question. I found if I run:
[user@host ~]# mysql --ssl-ca=/var/lib/mysql/ssl/ca-cert.pem --ssl-cert=/var/lib/mysql/ssl/client-cert.pem --ssl-key=/var/lib/mysql/ssl/client-key.pem -u replication -p -h 123.456.78.90
Enter password:
ERROR 2026 (HY000): SSL connection error: error:00000001:lib(0):func(0):reason(1)
So it appears that, for some reason, the SSL certificates have suddenly stopped working, though they haven't changed for months on both the master and slave.
Where should I turn next to figure out the problem?