I have got into slave lag gradually say every minute it lags around 5secs on slave.
So it started morning 10AM the Seconds_behind_master was 13. Now at 4:30 PM it has reached to 560.
No variable changed. Not much selects are running in there, its all regular ones. In fact we have stopped few applications thinking those could cause slowness due to selects.
This seems to be strange for me. This had never happened before.
I have below variables already set to minimize IO load on disk.
> innodb_flush_log_at_trx_commit=0
> innodb_flush_method = O_DIRECT
> sync_binlog=0
I also have 2 more slave nodes. They are performing real good. And they are used heavily by selects on app end.
Any thoughts or any one have run into similar issue before?
New Update
Something found in network.
Transferring 1 GB file to slave0
mannoj@master:~$ scp mysql-bin.001539 mannoj@12.168.4.7:/home/mannoj/
100% 1024MB 68.3MB/s 00:15
Transferring 1 GB file to slave1 mannoj@master:~$ scp mysql-bin.001539 mannoj@12.18.11.91:/home/mannoj/
100% 1024MB 170.7MB/s 00:06
Transferring 1 GB file to newslave000
mannoj@master:~$ scp mysql-bin.001539 mannoj@12.18.19.1:/home/mannoj/
100% 1024MB 204.8MB/s 00:05
The slave lag persist on slave0.
Final Update (I guess)>
I disabled Network consuming mysqlbinlog puller runs via cron. Seems to be draining seconds_behind_master from 700 to 490.