We recently set a master-slave replication with MariaDB 10.1.12 on RedHat 6.7 fully patched. Master and slave are powerful Dell R730 with 2 RAID arrays- 1) 2 x 15k SAS drives for OS in RAID1 2) 4 x Intel S3700 SSD in RAID10 solely for MySQL data files
~20GB database.
When idle, master loadavg is around 0.2-0.5 which is expected. Under load it barely goes above 1. At the same time the slave is cruising at around 5-6 loadavg even when idle.
iostat -N -x 2 on slave shows serious write activity to the SSD array:
avg-cpu: %user %nice %system %iowait %steal %idle
0.04 0.00 0.04 0.00 0.00 99.92
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
sda 0.00 1.00 0.00 2.50 0.00 28.00 11.20 0.00 1.80 0.60 0.15
sdb 0.00 175.50 0.00 97.50 0.00 2184.00 22.40 0.00 0.04 0.04 0.40
vgOS-swap 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
vgOS-root 0.00 0.00 0.00 2.00 0.00 16.00 8.00 0.01 3.00 0.75 0.15
vgDB-dbdata1 0.00 0.00 0.00 273.00 0.00 2184.00 8.00 0.02 0.09 0.02 0.45
vgOS-var 0.00 0.00 0.00 1.50 0.00 12.00 8.00 0.00 0.00 0.00 0.00
vgOS-u2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
avg-cpu: %user %nice %system %iowait %steal %idle
0.03 0.00 0.03 0.00 0.00 99.94
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdb 0.00 202.00 0.00 111.00 0.00 2504.00 22.56 0.01 0.05 0.04 0.45
vgOS-swap 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
vgOS-root 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
vgDB-dbdata1 0.00 0.00 0.00 313.00 0.00 2504.00 8.00 0.03 0.09 0.01 0.45
vgOS-var 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
vgOS-u2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
avg-cpu: %user %nice %system %iowait %steal %idle
0.02 0.00 0.04 0.00 0.00 99.94
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdb 0.00 205.00 0.00 109.50 0.00 2516.00 22.98 0.01 0.06 0.06 0.65
vgOS-swap 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
vgOS-root 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
vgDB-dbdata1 0.00 0.00 0.00 314.50 0.00 2516.00 8.00 0.01 0.03 0.02 0.70
vgOS-var 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
vgOS-u2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
avg-cpu: %user %nice %system %iowait %steal %idle
0.19 0.00 0.06 0.00 0.00 99.75
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdb 0.00 206.00 0.00 116.00 0.00 2576.00 22.21 0.00 0.04 0.04 0.45
vgOS-swap 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
vgOS-root 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
vgDB-dbdata1 0.00 0.00 0.00 322.00 0.00 2576.00 8.00 0.02 0.07 0.02 0.50
vgOS-var 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
vgOS-u2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00Same stats on master show way lower writes.
Same my.cnf is used on other master-slave setups too, with the only difference that they are running MySQL 5.1.73.
[client]
port = 3306
socket = /var/lib/mysql/mysql.sock
[mysqld_safe]
log-error=/var/log/mysql/error.log
pid-file=/var/run/mysqld/mysqld.pid
[mysqld]
datadir=/dbdata1/mysql
socket=/var/lib/mysql/mysql.sock
user=mysql
# Disabling symbolic-links is recommended to prevent assorted security risks
symbolic-links=0
tmpdir = /dbdata1/tmp
skip-external-locking
skip-name-resolve
# Charset and Collation
character-set-server = utf8
collation-server = utf8_unicode_ci
event_scheduler = 0
server-id = 2
log_bin = /dbdata1/mysql-bin.log
relay-log=mysql-relay-bin
relay-log-index=mysql-relay-bin.index
#log-slave-updates=1
read-only=1
replicate-ignore-table=asterisk.vicidial_admin_log
slave-skip-errors=all
#
# Instead of skip-networking the default is now to listen only on
# localhost which is more compatible and is not less secure.
bind-address = XXXXXXXXXXX
#
# * Fine Tuning
#
key_buffer_size = 8G
max_allowed_packet = 16M
thread_stack = 256K
thread_cache_size = 8
sort_buffer_size = 2M
read_buffer_size = 128k
read_rnd_buffer_size = 256k
join_buffer_size = 128k
auto-increment-increment = 1
auto-increment-offset = 1
# This replaces the startup script and checks MyISAM tables if needed
# the first time they are touched
myisam-recover = BACKUP
max_connections = 1024
max_connect_errors = 10
concurrent_insert = 2
connect_timeout = 10
wait_timeout = 180
net_read_timeout = 30
net_write_timeout = 30
back_log = 128
table_cache = 8092
tmp_table_size = 2G
max_heap_table_size = 2G
bulk_insert_buffer_size = 2G
open-files-limit = 24576
# Default Table Settings
#
# * Query Cache Configuration
#
query_cache_limit = 1M
query_cache_size = 16M
#
# * Logging
#
# Both location gets rotated by the cronjob.
# Be aware that this log type is a performance killer.
#log = /var/log/mysql/mysql.log
#
# Error logging goes to syslog. This is a Debian improvement :)
#
# * Replication
#
#
# Here you can see queries with especially long duration
slow_query_log = 1
slow_query_log_file = /var/log/mysql/slow.log
long_query_time = 2
#
# The following can be used as easy to replay backup logs or for replication.
# note: if you are setting up a replication slave, see README.Debian about
# other settings you may need to change.
expire_logs_days = 10
max_binlog_size = 100M
binlog_cache_size = 64K
#binlog_do_db = include_database_name
#binlog_ignore_db = include_database_name
binlog_format=mixed
sync_binlog = 0
slave_compressed_protocol = 0
#
# * InnoDB
#
# InnoDB is enabled by default with a 10MB datafile in /var/lib/mysql/.
# Read the manual for more InnoDB related options. There are many!
# You might want to disable InnoDB to shrink the mysqld process by circa 100MB.
#skip-innodb
## InnoDB Plugin Independent Settings
innodb_data_home_dir = /dbdata1/mysql
innodb_log_group_home_dir = /dbdata1/mysql
innodb_file_per_table
innodb_table_locks = true
innodb_lock_wait_timeout = 60
innodb_thread_concurrency = 64
innodb_commit_concurrency = 64
innodb_support_xa = true
innodb_buffer_pool_size = 128M
innodb_log_file_size = 10M
innodb_additional_mem_pool_size = 8M
innodb_data_file_path = ibdata1:10M:autoextend
innodb_flush_log_at_trx_commit = 1
innodb_log_buffer_size = 8M
#
# * Security Features
#
# Read the manual, too, if you want chroot!
# For generating SSL certificates I recommend the OpenSSL GUI "tinyca".
#
# ssl-ca=/etc/mysql/cacert.pem
# ssl-cert=/etc/mysql/server-cert.pem
# ssl-key=/etc/mysql/server-key.pem
[mysqldump]
quick
quote-names
max_allowed_packet = 16M
[mysql]
#no-auto-rehash # faster start of mysql but no tab completition
[myisamchk]
key_buffer = 16M
myisam_sort_buffer_size = 128M
myisam_max_sort_file_size = 2147483648
myisam_repair_threads = 1
myisam-recover = BACKUPWe've tried moving the MySQL data files from the SSD array to the SAS one to rule out possible issues with the SSD disks but that didn't change anything.
As soon as slave process is stopped (stop slave), everything goes back to normal.
Any ideas what might be causing these excessive writes on the slave when replication is on?