We are using 3 servers. Master Server1 syncs data to Server2 & Server3.
Server1 -- Debian -- MySQL 5.5.40
Server2 -- CentOS 6.6 -- MySQL 5.5.45
Server3 -- CentOS 6.6 -- MySQL 5.5.43
All 3 servers are :
25Core 32GB RAM CPU Intel(R) Xeon(R) CPU E5-2630 v2 @2.60GHz CPU Cores 6
Three servers works fine. In certain occasion not in any specific day time; CPU usage randomly goes up to 41% in 1 or 2 servers. Its happening once or twice a day for last 1 week. Here are my investigation.
- Leap Second -- we disabled ntp and date is synced now.
- Process list -- Not too many connection or suspicious query that can have such effect.
- Ran $mysql> show slave status \G .. Found no lock
- Once noticed "Waiting for master to send event" with time 14855 etc. Which we dropped as other times this value is negligible.
- The one I am suspecting is "Slave has read all relay log; waiting for the slave I/O thread to update it" . Where time is under 3 or 5. This is one common message that i noticed all the time. So I tried to set innodb_write_io_thread and innodb_read_io_thread to 8.
- After restarting mysql service everything gets stable right away. So, I cant pinpoint this to Web application.
I have looked into other posts in here and tried to optimise my.cnf file. My queries are:
- What I could do to investigate the issue other than mentioned above?
- Any ideas what could be the cause?
- Any Resources that I should read which could direct me to solve this issue.
Thanks.