today, 2016-10-20 , MySQL suddenly shutdown ,is very terror，I have 4 mysql instance , M-M and M-M , and the slave server die first ,the next day ,all die ,can somebody tell me why?
2016-10-19 20:38:55 46074 [Note] 'CHANGE MASTER TO executed'. Previous state master_host='43.230.89.240', master_port= 6305, master_log_file='mysql-bin.000005', master_log_pos= 6586005, master_bind=''. New state master_host='43.230.89.240', master_port= 6305, master_log_file='mysql-bin.000013', master_log_pos= 120, master_bind=''.
2016-10-19 20:39:42 46074 [Warning] Storing MySQL user name or password information in the master info repository is not secure and is therefore not recommended. Please consider using the USER and PASSWORD connection options for START SLAVE; see the 'START SLAVE Syntax' in the MySQL Manual for more information.
2016-10-19 20:39:42 46074 [Warning] Slave SQL: If a crash happens this configuration does not guarantee that the relay log info will be consistent, Error_code: 0
2016-10-19 20:39:42 46074 [Note] Slave SQL thread initialized, starting replication in log 'mysql-bin.000013' at position 120, relay log '/data/mysql_6305_binlog/mysql-relay-bin.000001' position: 4
2016-10-19 20:39:42 46074 [ERROR] Slave I/O: error connecting to master 'slave@43.230.89.240:6305' - retry-time: 60 retries: 1, Error_code: 1045
2016-10-19 20:40:42 46074 [Note] Slave I/O thread: connected to master 'slave@43.230.89.240:6305',replication started in log 'mysql-bin.000013' at position 120
2016-10-19 20:40:42 46074 [Warning] The master's UUID has changed, although this should not happen unless you have changed it manually. The old UUID was 09facb79-95be-11e6-a34d-1051722d44a9.
2016-10-20 10:20:03 46074 [Note] Slave: received end packet from server due to dump thread being killed on master. Dump threads are killed for example during master shutdown, explicitly by a user, or when the master receives a binlog send request from a duplicate server UUID <b673ffe4-9600-11e6-a500-246e9616fcc4> : Error
2016-10-20 10:20:03 46074 [Note] Slave I/O thread killed while reading event
2016-10-20 10:20:03 46074 [Note] Slave I/O thread exiting, read up to log 'mysql-bin.000013', position 3286071
2016-10-20 10:20:03 46074 [Note] /usr/local/mysql_6305/bin/mysqld: Normal shutdown
2016-10-20 10:20:03 46074 [Note] Giving 5 client threads a chance to die gracefully
2016-10-20 10:20:03 46074 [Note] Event Scheduler: Purging the queue. 0 events
2016-10-20 10:20:03 46074 [Note] Shutting down slave threads
2016-10-20 10:20:03 46074 [Note] Error reading relay log event: slave SQL thread was killed
2016-10-20 10:20:13 46074 [Note] Forcefully disconnecting 3 remaining clients
2016-10-20 10:20:13 46074 [Warning] /usr/local/mysql_6305/bin/mysqld: Forcing close of thread 941 user: 'bigoDb'
2016-10-20 10:20:13 46074 [Warning] /usr/local/mysql_6305/bin/mysqld: Forcing close of thread 17 user: 'bigoDb'
2016-10-20 10:20:13 46074 [Warning] /usr/local/mysql_6305/bin/mysqld: Forcing close of thread 19 user: 'bigoDb'
2016-10-20 10:20:13 46074 [Note] Binlog end
2016-10-20 10:20:13 46074 [Note] Shutting down plugin 'partition'
2016-10-20 10:20:13 46074 [Note] Shutting down plugin 'ARCHIVE'
2016-10-20 10:20:13 46074 [Note] Shutting down plugin 'BLACKHOLE'
2016-10-20 10:20:13 46074 [Note] Shutting down plugin 'INNODB_SYS_DATAFILES'
2016-10-20 10:20:13 46074 [Note] Shutting down plugin 'INNODB_SYS_TABLESPACES'
2016-10-20 10:20:13 46074 [Note] Shutting down plugin 'INNODB_SYS_FOREIGN_COLS'
2016-10-20 10:20:13 46074 [Note] Shutting down plugin 'INNODB_SYS_FOREIGN'
2016-10-20 10:20:13 46074 [Note] Shutting down plugin 'INNODB_SYS_FIELDS'
2016-10-20 10:20:13 46074 [Note] Shutting down plugin 'INNODB_SYS_COLUMNS'
2016-10-20 10:20:13 46074 [Note] Shutting down plugin 'INNODB_SYS_INDEXES'
2016-10-20 10:20:13 46074 [Note] Shutting down plugin 'INNODB_SYS_TABLESTATS'
2016-10-20 10:20:13 46074 [Note] Shutting down plugin 'INNODB_SYS_TABLES'
2016-10-20 10:20:13 46074 [Note] Shutting down plugin 'INNODB_FT_INDEX_TABLE'
2016-10-20 10:20:13 46074 [Note] Shutting down plugin 'INNODB_FT_INDEX_CACHE'
2016-10-20 10:20:13 46074 [Note] Shutting down plugin 'INNODB_FT_CONFIG'
2016-10-20 10:20:13 46074 [Note] Shutting down plugin 'INNODB_FT_BEING_DELETED'
2016-10-20 10:20:13 46074 [Note] Shutting down plugin 'INNODB_FT_DELETED'
2016-10-20 10:20:13 46074 [Note] Shutting down plugin 'INNODB_FT_DEFAULT_STOPWORD'
2016-10-20 10:20:13 46074 [Note] Shutting down plugin 'INNODB_METRICS'
2016-10-20 10:20:13 46074 [Note] Shutting down plugin 'INNODB_BUFFER_POOL_STATS'
2016-10-20 10:20:13 46074 [Note] Shutting down plugin 'INNODB_BUFFER_PAGE_LRU'
2016-10-20 10:20:13 46074 [Note] Shutting down plugin 'INNODB_BUFFER_PAGE'
2016-10-20 10:20:13 46074 [Note] Shutting down plugin 'INNODB_CMP_PER_INDEX_RESET'
2016-10-20 10:20:13 46074 [Note] Shutting down plugin 'INNODB_CMP_PER_INDEX'
2016-10-20 10:20:13 46074 [Note] Shutting down plugin 'INNODB_CMPMEM_RESET'
2016-10-20 10:20:13 46074 [Note] Shutting down plugin 'INNODB_CMPMEM'
2016-10-20 10:20:13 46074 [Note] Shutting down plugin 'INNODB_CMP_RESET'
2016-10-20 10:20:13 46074 [Note] Shutting down plugin 'INNODB_CMP'
2016-10-20 10:20:13 46074 [Note] Shutting down plugin 'INNODB_LOCK_WAITS'
2016-10-20 10:20:13 46074 [Note] Shutting down plugin 'INNODB_LOCKS'
2016-10-20 10:20:13 46074 [Note] Shutting down plugin 'INNODB_TRX'
2016-10-20 10:20:13 46074 [Note] Shutting down plugin 'InnoDB'
2016-10-20 10:20:13 46074 [Note] InnoDB: FTS optimize thread exiting.
2016-10-20 10:20:13 46074 [Note] InnoDB: Starting shutdown...
2016-10-20 10:20:15 46074 [Note] InnoDB: Shutdown completed; log sequence number 29179177
2016-10-20 10:20:15 46074 [Note] Shutting down plugin 'PERFORMANCE_SCHEMA'
2016-10-20 10:20:15 46074 [Note] Shutting down plugin 'CSV'
2016-10-20 10:20:15 46074 [Note] Shutting down plugin 'MEMORY'
2016-10-20 10:20:15 46074 [Note] Shutting down plugin 'MRG_MYISAM'
2016-10-20 10:20:15 46074 [Note] Shutting down plugin 'MyISAM'
2016-10-20 10:20:15 46074 [Note] Shutting down plugin 'sha256_password'
2016-10-20 10:20:15 46074 [Note] Shutting down plugin 'mysql_old_password'
2016-10-20 10:20:15 46074 [Note] Shutting down plugin 'mysql_native_password'
2016-10-20 10:20:15 46074 [Note] Shutting down plugin 'binlog'
2016-10-20 10:20:15 46074 [Note] /usr/local/mysql_6305/bin/mysqld: Shutdown complete
161020 10:20:15 mysqld_safe mysqld from pid file /usr/local/mysql_6305/mysql_6305.pid ended
and my.cnf is all the same
[root@localhost mysql_6305_log]# cat /data/mysql_6305/my.cnf
[client]
default-character-set = utf8mb4
port = 6305
socket = /usr/local/mysql_6305/mysql_6305.sock
[mysqld]
#lc_messages_dir = /usr/local/mysql_6305/share
#handlersocket
#loose_handlersocket_port = 49991
#loose_handlersocket_port_wr = 59991
#loose_handlersocket_threads = 16
#loose_handlersocket_threads_wr = 1
open_files_limit = 65535
#handlersocket_accept_balanc=16
#handlersocket_plain_secret = GeATtktjUoXv
#handlersocket_plain_secret_wr = GeATtktjUoXv
port = 6305
socket = /usr/local/mysql_6305/mysql_6305.sock
pid-file = /usr/local/mysql_6305/mysql_6305.pid
datadir=/data/mysql_6305
tmpdir=/data/mysql_6305
#charset
character-set-server=utf8mb4
key_buffer_size = 16M
max_allowed_packet = 32M
table_open_cache = 2000
sort_buffer_size = 2M
read_buffer_size = 2M
read_rnd_buffer_size = 8M
myisam_sort_buffer_size = 16M
thread_cache_size = 2000
query_cache_size = 0M
query_cache_type=0
thread_concurrency = 64
max_heap_table_size = 16M
bulk_insert_buffer_size = 16M
myisam_sort_buffer_size = 16M
myisam_max_sort_file_size = 16M
myisam_repair_threads = 1
myisam_recover
#connect
max_connections = 5000
max_connect_errors=999999
slow_launch_time=1
skip-name-resolve
init-connect="insert into auditdb.accesslog values(null,connection_id(),now(),user(),current_user());"
#log
log-error=/data/mysql_6305_log/error.log
general_log_file=/data/mysql_6305_log/general_log.log
binlog_format=row
log_bin = /data/mysql_6305_binlog/mysql-bin.log
relay_log = /data/mysql_6305_binlog/mysql-relay-bin.log
max_binlog_size = 32M
log_slave_updates
expire_logs_days = 7
slow_query_log_file=/data/mysql_6305_log/slow.log
slow_query_log=on
long_query_time=1
#slave
server-id = 31746305
#skip_slave_start
slave-skip-errors=1062
replicate_ignore_db=mysql
replicate_wild_ignore_table=mysql.%
#auto_increment_offset = 1
#auto_increment_increment = 2
slave_compressed_protocol = 1
binlog_ignore_db=auditdb
#isolation
transaction_isolation = READ-COMMITTED
#corefile
core-file
#innodb
innodb_data_home_dir = /data/mysql_6305
innodb_data_file_path = ibdata:100M:autoextend
innodb_log_group_home_dir = /data/mysql_6305
innodb_buffer_pool_size = 1G
innodb_additional_mem_pool_size = 8M
innodb_log_file_size = 64M
innodb_log_buffer_size = 32M
innodb_flush_log_at_trx_commit = 0
#innodb_lock_wait_timeout = 50
innodb_log_files_in_group=4
innodb_max_dirty_pages_pct = 90
innodb_open_files=2000
innodb_doublewrite=1
innodb_file_per_table
innodb_read_io_threads=32
innodb_write_io_threads=16
# innodb_thread_concurrency=16
innodb_purge_threads=1
#function
log_bin_trust_function_creators = 1
[mysqldump]
quick
max_allowed_packet = 1G
[mysql]
auto-rehash
[myisamchk]
key_buffer_size = 16M
sort_buffer_size = 16M
read_buffer = 2M
write_buffer = 2M
[mysqlhotcopy]
interactive-timeout
and this is another error log before normal shutdown
2016-10-20 11:56:19 57130 [Warning] 'user' entry '@localhost.localdomain' ignored in --skip-name-resolve mode.
2016-10-20 11:56:19 57130 [Warning] 'proxies_priv' entry '@ root@localhost.localdomain' ignored in --skip-name-resolve mode.
2016-10-20 11:56:19 57130 [Note] Event Scheduler: Loaded 0 events
2016-10-20 11:56:19 57130 [Note] /usr/local/mysql_6305/bin/mysqld: ready for connections.
Version: '5.6.34-log' socket: '/usr/local/mysql_6305/mysql_6305.sock' port: 6305 MySQL Community Server (GPL)
Status information:
Current dir: /data/mysql_6305/
Running threads: 3 Stack size: 262144
Current locks:
lock: 0x7fdc48063420:
lock: 0x7fdc48060ea0:
lock: 0x7fdc4805e4a0:
lock: 0x7fdc4805bfc0:
lock: 0x7fdc48059b50:
lock: 0x7fdc48057de8:
lock: 0x7fdc48048fe0:
lock: 0x7fdc48042870:
lock: 0x7fdc4803f720:
lock: 0x7fdc48039680:
lock: 0x7fdc48033fe0:
lock: 0x7fdc4802cc80:
lock: 0x7fdc48027ed0:
lock: 0x7fdc48023310:
lock: 0x7fdc4801f050:
lock: 0x7fdc4801b0e8:
lock: 0x7fdc4800f7d0:
lock: 0x7fdc68043630:
lock: 0x7fdc68038eb0:
lock: 0x7fdc68035800:
lock: 0x7fdc68030850:
lock: 0x7fdc6802d200:
lock: 0x7fdc6800dc00:
lock: 0x7fdc6800aa50:
lock: 0x32daa50:
lock: 0x31db0a0:
lock: 0x31d7b50:
lock: 0x315dc70:
lock: 0x31c55f0:
lock: 0x31b72a0:
lock: 0x31acf10:
lock: 0x3190ff0:
lock: 0x31795e0:
lock: 0x3171320:
lock: 0x3167c20:
lock: 0x1385020:
lock: 0x1384f40:
lock: 0x1383f80:
lock: 0x1383ea0:
lock: 0x1383dc0:
lock: 0x1384140:
lock: 0x1382380:
lock: 0x1384e60:
lock: 0x1382a80:
lock: 0x1382d20:
lock: 0x13829a0:
lock: 0x1382c40:
lock: 0x13828c0:
lock: 0x1382b60:
lock: 0x1383260:
lock: 0x1383180:
lock: 0x13830a0:
lock: 0x13827e0:
lock: 0x1382540:
lock: 0x1382700:
lock: 0x1382460:
lock: 0x1382620:
lock: 0x1382fc0:
lock: 0x1382ee0:
lock: 0x1382e00:
lock: 0x1384ae0:
lock: 0x1384d80:
lock: 0x1384ca0:
lock: 0x1384bc0:
lock: 0x1384760:
lock: 0x1384680:
lock: 0x13845a0:
lock: 0x13844c0:
lock: 0x13843e0:
lock: 0x1384920:
lock: 0x1384300:
lock: 0x1384220:
lock: 0x1384840:
lock: 0x1384060:
lock: 0x1383c00:
lock: 0x1383ce0:
lock: 0x1383b20:
lock: 0x1383a40:
lock: 0x13836c0:
lock: 0x1383960:
lock: 0x1383880:
lock: 0x13835e0:
lock: 0x13837a0:
lock: 0x1383500:
lock: 0x1383420:
lock: 0x1383340:
lock: 0x1384a00:
Key caches:
default
Buffer_size: 16777216
Block_size: 1024
Division_limit: 100
Age_limit: 300
blocks used: 0
not flushed: 0
w_requests: 0
writes: 0
r_requests: 0
reads: 0
handler status:
read_key: 25319
read_next: 54
read_rnd 486
read_first: 24634
write: 276051
delete 0
update: 8039
Table status:
Opened tables: 94
Open tables: 87
Open files: 57
Open streams: 0
Alarm status:
Active alarms: 0
Max used alarms: 0
Next alarm time: 0
Events status:
LLA = Last Locked At LUA = Last Unlocked At
WOC = Waiting On Condition DL = Data Locked
Event scheduler status:
State : INITIALIZED
Thread id : 0
LLA : n/a:0
LUA : n/a:0
WOC : NO
Workers : 0
Executed : 0
Data locked: NO
Event queue status:
Element count : 0
Data locked : NO
Attempting lock : NO
LLA : init_queue:139
LUA : init_queue:149
WOC : NO
Next activation : never
2016-10-20 16:37:44 57130 [Warning] 'user' entry 'root@localhost.localdomain' ignored in --skip-name-resolve mode.
2016-10-20 16:37:44 57130 [Warning] 'user' entry '@localhost.localdomain' ignored in --skip-name-resolve mode.
2016-10-20 16:37:44 57130 [Warning] 'proxies_priv' entry '@ root@localhost.localdomain' ignored in --skip-name-resolve mode.
2016-10-20 16:37:44 57130 [Note] /usr/local/mysql_6305/bin/mysqld: Normal shutdown
2016-10-20 16:37:44 57130 [Note] Giving 3 client threads a chance to die gracefully
2016-10-20 16:37:44 57130 [Note] Event Scheduler: Purging the queue. 0 events
2016-10-20 16:37:44 57130 [Note] Shutting down slave threads
2016-10-20 16:37:54 57130 [Note] Forcefully disconnecting 1 remaining clients
2016-10-20 16:37:54 57130 [Warning] /usr/local/mysql_6305/bin/mysqld: Forcing close of thread 2 user: 'bigoDb'
2016-10-20 16:37:54 57130 [Note] Binlog end
2016-10-20 16:37:54 57130 [Note] Shutting down plugin 'partition'
2016-10-20 16:37:54 57130 [Note] Shutting down plugin 'ARCHIVE'
2016-10-20 16:37:54 57130 [Note] Shutting down plugin 'BLACKHOLE'
2016-10-20 16:37:54 57130 [Note] Shutting down plugin 'INNODB_SYS_DATAFILES'
2016-10-20 16:37:54 57130 [Note] Shutting down plugin 'INNODB_SYS_TABLESPACES'
2016-10-20 16:37:54 57130 [Note] Shutting down plugin 'INNODB_SYS_FOREIGN_COLS'
2016-10-20 16:37:54 57130 [Note] Shutting down plugin 'INNODB_SYS_FOREIGN'
2016-10-20 16:37:54 57130 [Note] Shutting down plugin 'INNODB_SYS_FIELDS'
2016-10-20 16:37:54 57130 [Note] Shutting down plugin 'INNODB_SYS_COLUMNS'
2016-10-20 16:37:54 57130 [Note] Shutting down plugin 'INNODB_SYS_INDEXES'
2016-10-20 16:37:54 57130 [Note] Shutting down plugin 'INNODB_SYS_TABLESTATS'
2016-10-20 16:37:54 57130 [Note] Shutting down plugin 'INNODB_SYS_TABLES'
2016-10-20 16:37:54 57130 [Note] Shutting down plugin 'INNODB_FT_INDEX_TABLE'
2016-10-20 16:37:54 57130 [Note] Shutting down plugin 'INNODB_FT_INDEX_CACHE'
2016-10-20 16:37:54 57130 [Note] Shutting down plugin 'INNODB_FT_CONFIG'
2016-10-20 16:37:54 57130 [Note] Shutting down plugin 'INNODB_FT_BEING_DELETED'
2016-10-20 16:37:54 57130 [Note] Shutting down plugin 'INNODB_FT_DELETED'
2016-10-20 16:37:54 57130 [Note] Shutting down plugin 'INNODB_FT_DEFAULT_STOPWORD'
2016-10-20 16:37:54 57130 [Note] Shutting down plugin 'INNODB_METRICS'
2016-10-20 16:37:54 57130 [Note] Shutting down plugin 'INNODB_BUFFER_POOL_STATS'
2016-10-20 16:37:54 57130 [Note] Shutting down plugin 'INNODB_BUFFER_PAGE_LRU'
2016-10-20 16:37:54 57130 [Note] Shutting down plugin 'INNODB_BUFFER_PAGE'
2016-10-20 16:37:54 57130 [Note] Shutting down plugin 'INNODB_CMP_PER_INDEX_RESET'
2016-10-20 16:37:54 57130 [Note] Shutting down plugin 'INNODB_CMP_PER_INDEX'
2016-10-20 16:37:54 57130 [Note] Shutting down plugin 'INNODB_CMPMEM_RESET'
2016-10-20 16:37:54 57130 [Note] Shutting down plugin 'INNODB_CMPMEM'
2016-10-20 16:37:54 57130 [Note] Shutting down plugin 'INNODB_CMP_RESET'
2016-10-20 16:37:54 57130 [Note] Shutting down plugin 'INNODB_CMP'
2016-10-20 16:37:54 57130 [Note] Shutting down plugin 'INNODB_LOCK_WAITS'
2016-10-20 16:37:54 57130 [Note] Shutting down plugin 'INNODB_LOCKS'
2016-10-20 16:37:54 57130 [Note] Shutting down plugin 'INNODB_TRX'
2016-10-20 16:37:54 57130 [Note] Shutting down plugin 'InnoDB'
2016-10-20 16:37:54 57130 [Note] InnoDB: FTS optimize thread exiting.
2016-10-20 16:37:54 57130 [Note] InnoDB: Starting shutdown...
2016-10-20 16:37:56 57130 [Note] InnoDB: Shutdown completed; log sequence number 30711082
2016-10-20 16:37:56 57130 [Note] Shutting down plugin 'PERFORMANCE_SCHEMA'
2016-10-20 16:37:56 57130 [Note] Shutting down plugin 'CSV'
2016-10-20 16:37:56 57130 [Note] Shutting down plugin 'MEMORY'
2016-10-20 16:37:56 57130 [Note] Shutting down plugin 'MRG_MYISAM'
2016-10-20 16:37:56 57130 [Note] Shutting down plugin 'MyISAM'
2016-10-20 16:37:56 57130 [Note] Shutting down plugin 'sha256_password'
2016-10-20 16:37:56 57130 [Note] Shutting down plugin 'mysql_old_password'
2016-10-20 16:37:56 57130 [Note] Shutting down plugin 'mysql_native_password'
2016-10-20 16:37:56 57130 [Note] Shutting down plugin 'binlog'
2016-10-20 16:37:56 57130 [Note] /usr/local/mysql_6305/bin/mysqld: Shutdown complete
and I discover that in the mysql database document,there is no host.frm host.idb