I should emphasize that my question is in regard to CP (Consistent and Partition Tolerant) systems.

A lot of widely used NoSQL databases such as Mongo and Redis seem to fall in the CP category. And in addition to that, RDBMS's such as SQL Server and MySQL can also be configured to operate in a CP multi-master mode.

Two arguments I could find on the Internet are:

1. Load balancing write queries
2. Decreasing downtime in case of failure of a Master

I have doubts about the first reason because as far as I understand, in order to maintain data consistency between these Master replicas, the write operation should take place in every Master. If so, that will create the same burden for all the Masters plus the overhead associated for synchronization.

What are the solid reasons for such systems? Or am I missing something?

I am configuring replication from EC2 master to Amazon RDS instance.

After starting the slave, I don't see any errors but I see slave_IO:

_thread is connecting. 

Master version:5.6.23
Slave Version:5.6.19

show slave status \G

mysql> show slave status \G
*************************** 1. row ***************************
               Slave_IO_State: Connecting to master
                  Master_Host: XXXXXXXXXXX
                  Master_User: XXXXXX
                  Master_Port: 3306
                Connect_Retry: 60
              Master_Log_File: cli-bin.000032
          Read_Master_Log_Pos: 713
               Relay_Log_File: relaylog.000001
                Relay_Log_Pos: 4
        Relay_Master_Log_File: cli-bin.000032
             Slave_IO_Running: Connecting
            Slave_SQL_Running: Yes
              Replicate_Do_DB:
          Replicate_Ignore_DB:
           Replicate_Do_Table:
       Replicate_Ignore_Table: mysql.plugin,innodb_memcache.cache_policies,mysql.rds_sysinfo,mysql.rds_replication_status,mysql.rds_history,innodb_memcache.config_options
      Replicate_Wild_Do_Table:
  Replicate_Wild_Ignore_Table:
                   Last_Errno: 0
                   Last_Error:
                 Skip_Counter: 0
          Exec_Master_Log_Pos: 713
              Relay_Log_Space: 618
              Until_Condition: None
               Until_Log_File:
                Until_Log_Pos: 0
           Master_SSL_Allowed: No
           Master_SSL_CA_File:
           Master_SSL_CA_Path:
              Master_SSL_Cert:
            Master_SSL_Cipher:
               Master_SSL_Key:
        Seconds_Behind_Master: NULL
Master_SSL_Verify_Server_Cert: No
                Last_IO_Errno: 0
                Last_IO_Error:
               Last_SQL_Errno: 0
               Last_SQL_Error:
  Replicate_Ignore_Server_Ids:
             Master_Server_Id: 0
                  Master_UUID:
             Master_Info_File: mysql.slave_master_info
                    SQL_Delay: 0
          SQL_Remaining_Delay: NULL
      Slave_SQL_Running_State: Slave has read all relay log; waiting for the slave I/O thread to update it
           Master_Retry_Count: 86400
                  Master_Bind:
      Last_IO_Error_Timestamp:
     Last_SQL_Error_Timestamp:
               Master_SSL_Crl:
           Master_SSL_Crlpath:
           Retrieved_Gtid_Set:
            Executed_Gtid_Set:
                Auto_Position: 0
1 row in set (0.00 sec)

show global variables

mysql> show global variables like '%old_passwords%';
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| old_passwords | 0     |
+---------------+-------+
1 row in set (0.01 sec)

mysql> show global variables like '%secure_auth%';
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| secure_auth   | OFF   |
+---------------+-------+
1 row in set (0.00 sec)
====================

The problem is Slave_IO_state is connecting and Slave_io_thread is connecting state but replication is not happening.

I've set up a MySQL "cluster" (four CentOS 7 servers running MySQL 5.7.19) using Group Replication in multi-primary mode, but I can't get it to accept writes from more than one thread at a time. This mode is recommended only for "advanced users", according to Oracle, so I guess that's the source of my troubles.

The group that I've set up works: I can write and read from it, it stays in sync, all good. However, we have a load test (in Java, running on Tomcat) that I'm running on the cluster, that consistently fails when launched with more than one thread. This load test runs a variety of transactions in as many threads as wanted as fast as it can towards a single node. What happens is that the transactions result in java.sql.SQLException: Plugin instructed the server to rollback the current transaction.. (This is, as far as I can gather, what is printed any time the group replication plugin has determined that some transaction must be rolled back for whatever reason). This eventually kills all but one thread, which continues happily until completion. The odd thing is that this load test is made to never create contention on any row; each thread gets its own set of rows to manipulate. Stopping the group replication plugin or running in single-primary mode fixes the issue, allowing me to run concurrent threads with write transactions.

Only having one writer at a time would be unacceptable in production, so this is a showstopper.

I've tried all the isolation levels (including read-uncommitted). I've tried running the appliers in parallel. I've read the requirements and limitations in particular and the entire group replication dev documentation from Oracle in general. I've tried reading bad translations of Chinese open source forums... No luck.

Has anyone gotten this to work, or knows how to?

EDIT: It is possible to run more than one thread against the same server, if the transactions are timed so that they interleave. That is, more than one connection can execute transactions, but only one can execute a transaction at any given point in time, otherwise one of the transactions will fail.

EDIT: Clarifying based on kind input from Matt Lord:

"Perhaps the writes being executed by your benchmark/load test are against a table with cascading FKs?" No, the output from grep --perl-regexp "ON DELETE CASCADE|ON UPDATE CASCADE|ON DELETE SET NULL|ON UPDATE SET NULL|ON DELETE SET DEFAULT|ON UPDATE SET DEFAULT" mysqldump_gr.sql -ni (where mysqldump_gr.sql is the result of mysqldump -u root -pvisa --triggers --routines --events --all-databases > mysqldump_gr.sql) results in one huge text insert into mysql.help_topic.

"[Can you give me a] MySQL error log snippet covering the relevant time period from the node(s) you're executing writes against[?]" As weird as it sounds, this varies. Either there is no output to the error log during the test or there are lines like this one: [Note] Aborted connection 1077 to db: 'mydb' user: 'user' host: 'whereISendTransactionsFrom' (Got an error reading communication packets). I didn't write about this error message because I thought it was just a one-off the first time we tested and none of the google results had anything to do with GR, but now I did another test and here it is again...

"[Can you give me] A basic definition of the load test: schema, queries, write pattern[?] (e.g. is each benchmark/client thread being executed against a different mysqld server?)" Unfortunately that's proprietary, but I can reiterate some info from above: The test is executed against a single node (i.e. a single server). Each thread gets its own rows to manipulate.

"[Can you give me] The my.cnf being used on the mysql instances[?]" I've tried with two different ones, though with many similarities due to requirements. This is the latest one, anonymized a bit:

[mysql]
port                           = 3306
socket                         = /var/lib/mysql/mysql.sock
[mysqld]
port                           = 3306
socket                         = /var/lib/mysql/mysql.sock
transaction_isolation          = READ-UNCOMMITTED
explicit_defaults_for_timestamp= ON
user                           = mysql
default-storage-engine         = InnoDB
socket                         = /var/lib/mysql/mysql.sock
pid-file                       = /var/lib/mysql/mysql.pid
bind-address                   = 0.0.0.0
skip-host-cache
secure-file-priv               = ""
report_host                    = "realIpAddressHere"
datadir                        = /var/lib/mysql/
log-bin                        = /var/lib/mysql/mysql-bin
relay-log                      = /var/lib/mysql/relay-bin
server-id                      = 59331200
server_id                      = 59331200
auto_increment_increment       = 10
auto_increment_offset          = 1
replicate-ignore-db            = mysql
slave-skip-errors              = 1032,1062
master-info-repository         = TABLE
relay-log-info-repository      = TABLE
binlog_checksum                = NONE
gtid_mode                      = ON
enforce_gtid_consistency       = ON
log_slave_updates              = ON
log_bin                        = binlog
binlog_format                  = ROW
transaction_write_set_extraction         = XXHASH64
loose-group_replication_group_name       = "aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa"
loose-group_replication_start_on_boot    = off
loose-group_replication_local_address    = "localAddressHere"
loose-group_replication_group_seeds      = "groupSeedsHere"
loose-group_replication_bootstrap_group  = off
loose-group_replication_single_primary_mode = OFF
loose-group_replication_enforce_update_everywhere_checks = ON
disabled_storage_engines="MyISAM,BLACKHOLE,FEDERATED,ARCHIVE,MEMORY"
loose-group_replication_ip_whitelist="ipRangeHere"
slave_parallel_workers         = 1024
slave_transaction_retries      = 18446744073709551615
slave_skip_errors              = ddl_exist_errors
loose-group_replication_gtid_assignment_block_size = 1024
log-error                      = /var/lib/mysql/mysql-error.log
log-queries-not-using-indexes  = 0
slow-query-log                 = 1
slow-query-log-file            = /var/lib/mysql/mysql-slow.log
event_scheduler=ON
loose-group_replication_single_primary_mode = OFF
loose-group_replication_enforce_update_everywhere_checks = ON

We do not have a MySQL Enterprise subscription.

We currently have a NodeJS program collecting data from a socket that gets sent realtime information from one our other services.

This data is then stored in a Database by the afore mentioned instance. This process simply listens for the stream. Parses the objects out and then updates the database with the information.

Another service can then simply read the data from the database. Which is what our users poll for the info.

My question is:

How would I make sure 2 instances of this code is running and kept running in 2 separate data centres ? So if one instance goes down the other simply starts updating the database. While we fix the other process.

Here what I was thinking:

1. Was thinking of having 2 PosgresSQL servers running, one in each data centre. Running in Master and Standby mode. This will enabled the web services in both data centres to read the data from the database but not update it while the master is still alive. Giving me the only one update that I want.

2. But the other problem is what if the NodeJS instances dies. I then have no way of knowing ? I could implement a heartbeat system but there has to be a better ?

3. Also considered running a Rabbit MQ service with a queues, but that gives me a single point of failure.

Been thinking around this for a while but can't seem to find a architecture that would work. Any advice ?

I already have Master Slave configuration on AWS RDS. We want to upgrade 5.5 to 5.6.23. ..What should be correct sequence to Upgrade ?

1. Upgrade Master ( Using Modify Instance ) --> Upgrade Slave
2. Upgrade Master ( Using Modify Instance ) --> Create a new Read Replica ?

I have created 3 Digitalocean droplet. By default, I choose Ubuntu 18.06 and MongoDB 4. Here I have 3 droplets by default MongoDB config and all are up. I have access to "mongo" shell for all of them.

Now I want to run a replica-set setting by this code:

rs.initiate( 
    {_id : "rs0",
    members: [
        { _id: 0, host: "20.30.40.50:27017" },
        { _id: 1, host: "20.30.40.51:27017" },
        { _id: 2, host: "20.30.40.52:27017" }
    ]
})

In this config, I just told MongoDB that rune the replica set and it retrieves me the error

no replies config has been received

I did not add any bindIp yet also when I added bindIp, I could not start MongoDB again. I put in mongo.conf like this:

bindIp: 127.0.0.1,20.30.40.51,20.30.40.52

Also, There is a private network between these 3 droplets by IP example: 10.10.1.1 Can I take advantage of this private IP to make it easier and safer?

I have the following replication topology:

DB1 (MySQL 5.5) -> DB2 (MySQL 5.6, explicit_defaults_for_timestamp = 1) -> DB3 (MySQL 5.6, explicit_defaults_for_timestamp = 1)

- "date" field:

`date` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,

- DB3 replication error:

[ERROR] Slave SQL: Error 'Column 'date' cannot be null' on query. Default database: 'test'. Query: 'INSERT INTO test_log VALUES (null,'12345',12345,'test','saved')', Error_code: 1048

The reason why DB3 is failing is explained here:

No TIMESTAMP column is assigned the DEFAULT CURRENT_TIMESTAMP or ON UPDATE CURRENT_TIMESTAMP attributes automatically. Those attributes must be explicitly specified.

I would like to understand why DB2 is working fine, I guess that's because it's replicating from MySQL 5.5 but what settings are responsible for this?

Update Wed 1 Oct 09:34:03 BST 2014:

Table definition match on all three servers:

mysql> SHOW CREATE TABLE test_log\G
*************************** 1. row ***************************
       Table: feedback_log
Create Table: CREATE TABLE `feedback_log` (
  `date` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
  `order_ref` varchar(32) NOT NULL,
  `id` int(11) NOT NULL,
  `version` varchar(12) NOT NULL,
  `event` varchar(60) NOT NULL,
  KEY `order_ref` (`order_ref`),
  KEY `id` (`id`),
  KEY `version` (`version`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8
1 row in set (0.00 sec)

SQL_MODE shouldn't be the case here:

• DB1: None
• DB2, DB3: NO_ENGINE_SUBSTITUTION

Summary:

I can't run this query manually on both slaves (DB1, DB2) but it's replicated successfully on DB2:

mysql [5.6.20-68.0-log]> INSERT INTO test_log VALUES (null,'12345',12345,'test','saved')';
ERROR 1048 (23000): Column 'date' cannot be null

Another quick test showing this behaviour:

DB1

mysql [5.5.39-log]> CREATE TABLE t1 (date TIMESTAMP);
Query OK, 0 rows affected (0.20 sec)

mysql [5.5.39-log]> SHOW CREATE TABLE t1\G
*************************** 1. row ***************************
       Table: t1
Create Table: CREATE TABLE `t1` (
  `date` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP
) ENGINE=InnoDB DEFAULT CHARSET=utf8
1 row in set (0.00 sec)

DB2

mysql [5.6.20-68.0-log]> SELECT @@explicit_defaults_for_timestamp;
+-----------------------------------+
| @@explicit_defaults_for_timestamp |
+-----------------------------------+
|                                 1 |
+-----------------------------------+
1 row in set (0.00 sec)

mysql [5.6.20-68.0-log]> SHOW CREATE TABLE t1\G
*************************** 1. row ***************************
       Table: t1
Create Table: CREATE TABLE `t1` (
  `date` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP
) ENGINE=InnoDB DEFAULT CHARSET=utf8
1 row in set (0.00 sec)

DB3

mysql [5.6.20-68.0-log]> SELECT @@explicit_defaults_for_timestamp;
+-----------------------------------+
| @@explicit_defaults_for_timestamp |
+-----------------------------------+
|                                 1 |
+-----------------------------------+
1 row in set (0.04 sec)

mysql [5.6.20-68.0-log]> SHOW CREATE TABLE t1\G
*************************** 1. row ***************************
       Table: t1
Create Table: CREATE TABLE `t1` (
  `date` timestamp NULL DEFAULT NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8
1 row in set (0.00 sec)

I have 2 hbase cluster named A and B , each cluster have 3 machines, i want to enable replication between the 2 cluster, here is the step

Add config to hbase-site.xml

<property><name>hbase.replication</name><value>true</value></property>

In master cluster A, add peer

add_peer '2',"wxmaster1,wxmaster2,wxslave1:2181:/hbase2"

this return success, as I can see in log , cluster A master log have init the session connection to cluster B regionservers.

Create the same table in cluster A and B

create 'test_replication', {NAME => 'cf', REPLICATION_SCOPE => '1', VERSIONS => '2', KEEP_DELETED_CELLS => 'TRUE'}

Then I think everything should be ok, any change at cluster A table 'test_replication' will copy to cluster B, while it not, and no any error log.

Here is the results for status 'replication'

2017-02-16 18:31:34,570 WARN  [regionserver/wxmaster2/172.16.178.78:16021.logRoller] regionserver.ReplicationSource: Queue size: 3 exceeds value of replication.source.log.queue.warn: 2
2017-02-16 18:31:34,572 INFO  [regionserver/wxmaster2/172.16.178.78:16021.logRoller] wal.FSHLog: Archiving hdfs://cluster1/hbase/WALs/wxmaster2,16021,1487233892284/wxmaster2%2C16021%2C1487233892284.default.1487233894071 to hdfs://cluster1/hbase/oldWALs/wxmaster2%2C16021%2C1487233892284.default.1487233894071
2017-02-16 18:31:34,586 INFO  [regionserver/wxmaster2/172.16.178.78:16021.logRoller] wal.FSHLog: Archiving hdfs://cluster1/hbase/WALs/wxmaster2,16021,1487233892284/wxmaster2%2C16021%2C1487233892284.default.1487237494395 to hdfs://cluster1/hbase/oldWALs/wxmaster2%2C16021%2C1487233892284.default.1487237494395
2017-02-16 18:36:33,914 INFO  [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=1.65 MB, freeSize=1.57 GB, max=1.57 GB, blockCount=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,evictions=749, evicted=0, evictedPerRun=0.0
2017-02-16 18:36:34,394 INFO  [wxmaster2:16021Replication Statistics #0] regionserver.Replication: Normal source for cluster 2: Total replicated edits: 0, currently replicating from: null at position: 0

anybody can help me?

And, I am using

hadoop-2.5.2
hbase-1.0.1
zookeeper-3.4.6

Our MSrepl_errors table is around 15GB and got records from the beginning of time. What is the correct way of cleaning up this table. Do we simply delete from it?

Edit The distribution agent clean up is already running.

Cheers

I successfully generated the snapshot, but subscription is not working correctly. Either the synchronization takes too long to respond or it simply displays the text shown in below attached screenshot.

Here is the successful snapshot generation

It is only retrying and not giving the end result.

I want to setup transactional replication on sql server. By following the instructions of the ms replication preparation tutorial and the ms replication configuration tutorial, everything works fine.

My question is: Why all these 4 users? Will there be any trouble if instead of 4 I only use one windows user/db login for every step?

In the Google CloudSQL (MySQL) console, I can read:

Automated backups required for binary logging.

This requirement looks strange to me. I do not understand why automated backups are required for binary logging. I see use-cases where I might want replication (and failover, both needing binary logging) without automated backups.

Does anyone know why this requirement exists ?

My company is trying to implement a MySQL failover mechanism, to achieve higher availability in our webservices tier - we commercialize a SaaS solution. To that end we have some low-end VMs scattered through different geographical locations, each containing a MySQL 5.5 server with several DBs, that for the time being are merely slave-replicating from the production server - the objective up until now was just checking the latency and general resilience of MySQL replication.

The plan however is to add a Master-Master replication environment between two servers in two separate locations, and these two instances would handle all the DB writes. The idea wouldn't necessarily imply concurrency; rather the intention is having a single one of the instances handling the writes, and upon a downtime situation using a DNS Failover service to direct the requests to the secondary server. After the primary comes back online, the b-log generated in the meantime in the secondary would be replicated back, and the DNS Failover restored the requests back to the first one.

I am not an experienced administrator, so I'm asking for your own thoughts and experiences. How wrong is this train of thought? What can obviously go wrong? Are there any much better alternatives? Bash away!

Thanks!

We have a PROD DB which replicates into a slave DB using mixed replication. We want to add a trigger so that a row is added to our DW when a row is INSERTed into table_a (on master). The issue is that this INSERT is coming through using Row-based replication and the trigger (which is on table_a on slave) is not firing. We need to have the trigger on the slave table as that is where our DW is.

Looking around online it looks like this should work if statement-based replication is used. Is it possible to force the INSERT to table A to be processed as statement-based replication? Or is there any other way we can achieve this?

The INSERT itself is deterministic as is the trigger. We are using MySQL 5.6.

If you need any other information please let me know.

Thanks, Martin

I'm trying to do a point-in-time-recovery of our production database, but when reading the binary logs after restoring the dump, I get ERROR 1062 (23000) at line x in file: 'binlogs_file.sql': Duplicate entry 'y' for key 'PRIMARY'. I've double checked, and it seems insert statements in the binary logs already exists in the dump file, and consequently in the newly restored test database.

This is my mysqldump statement that runs every morning on our production server:

echo "SET autocommit=0;" >> backup_file.sql
echo "SET unique_checks=0;" >> backup_file.sql
echo "SET foreign_key_checks=0;" >> backup_file.sql
mysqldump --flush-logs --quick --single-transaction --master-data=2 --force --  routines  <databese_name> >> backup_file.sql`
echo "COMMIT;" >> backup_file.sql
echo "SET unique_checks=1;" >> backup_file.sql
echo "SET foreign_key_checks=1;" >> backup_file.sql

I copy the latest dump to the test server (which is a snapshot of our production server, only three weeks old). After restoring the test db, I retrieve what binlog file to start from, with this command:

cat backup_file.sql | grep 'CHANGE MASTER TO MASTER_LOG_FILE'

and this is returned:

CHANGE MASTER TO MASTER_LOG_FILE='mysql-bin.006502', MASTER_LOG_POS=154;

I copy the mysql-bin.006502 file, as well as all the succeeding binlog files from production server, and create one .sql file out of them all (note that the position is superfluous, as --flush-logs is part of the mysqldump statement)

mysqlbinlog mysql-bin.006502 > binlogs_file.sql
mysqlbinlog mysql-bin.006503 >> binlogs_file.sql
mysqlbinlog mysql-bin.006504 >> binlogs_file.sql 

Next step is to run the binlogs_file.sql against the db.

mysql -u <db_user> -p -e "source binlogs_file.sql"

Then the error occur:

ERROR 1062 (23000) at line x in file: 'binlogs_file.sql': Duplicate entry 'y' for key 'PRIMARY'

We're running MySql 5.7.19 on both production server, and test server (both Debian 8.10). These are the binlog related variables on the production server:

binlog_format                   = mixed
binlog_group_commit_sync_delay   =   0                                                                                                                              
binlog_group_commit_sync_no_delay_count = 0 

What am I doing wrong? Why the inconsistency?

I should emphasize that my question is in regard to CP (Consistent and Partition Tolerant) systems.

A lot of widely used NoSQL databases such as Mongo and Redis seem to fall in the CP category. And in addition to that, RDBMS's such as SQL Server and MySQL can also be configured to operate in a CP multi-master mode.

Two arguments I could find on the Internet are:

1. Load balancing write queries
2. Decreasing downtime in case of failure of a Master

I have doubts about the first reason because as far as I understand, in order to maintain data consistency between these Master replicas, the write operation should take place in every Master. If so, that will create the same burden for all the Masters plus the overhead associated for synchronization.

What are the solid reasons for such systems? Or am I missing something?

I have successfully created replication using GTID_MODE. It works perfectly. Now I need to setup automatic failover feature in it. I have run the following command.

mysqlfailover --master=root:abc@10.24.184.12:3306 --discover-slaves-login=root:abc

I have got the following results. No slave is listed.

MySQL Replication Failover Utility
Failover Mode = auto     Next Interval = Tue May

Master Information
------------------
Binary Log File   Position  Binlog_Do_DB  Binlog
mysql-bin.000016  9568

GTID Executed Set
8fe8b710-cd34-11e4-824d-fa163e52e544:1-1143

Replication Health Status
0 Rows Found.
Q-quit R-refresh H-health G-GTID Lists U-UUIDs U

But when I execute the mysqlrplcheck and mysqlrplshow commands, the slave is listed.

Is this normal?

Web Application is High school record management system (HSRMS). Users of the application are

1. High school management (Normal Users)
2. Data verifying users (Admin)

User group 2 will allow the User group 1 to submit data periodically and then user group 2 will validate it and ask for corrections. Hope the system flow is quite clear.

Replication Plan (Proposed)

I have setup a proposed transactional replication as

• Publisher (Server A)
• Distributor (Server B)
• Subscriber (Server(s) i.e C,D,E etc).

Note: All the servers will be in the same area/location.

What i understood is that the publisher should be the database server which should be connected to the HSRMS application, the distributor contains the transactions(updates) and is responsible to carry forward to subscribers, but i need to record a server status on any agent which should notify the sysadmin to update their DB server address in case of server failure (due to any reason) and the subscriber must be responding meanwhile.

1. Is my approach and understanding correct or i need to dive deep into it?
2. Is there any mechanism to keep the application server aware of the failure and act accordingly?
3. Dynamic Server connection upon failure of any db server? i.e Instead of sys admin the server connection should be automatically mapped to the active running db server?
4. What could be the role of subscribers if the application is just connected to the Server A i.e Publication for updates? Are subscribers there only for the DB backup and will not interact with the web application?

The migration our App runs (full source):

ALTER TABLE permission
  ADD COLUMN path VARCHAR(255);
UPDATE permission
SET path = (select name
            from credential
            where uuid = permission.credential_uuid);

(..)

This is the error in App log

2018-10-09T09:42:00.604Z [main] ....  INFO --- VersionPrinter: Flyway Community Edition 5.0.7 by Boxfuse
2018-10-09T09:42:00.615Z [main] ....  INFO --- HikariDataSource: HikariPool-1 - Starting...
2018-10-09T09:42:00.812Z [main] ....  INFO --- HikariDataSource: HikariPool-1 - Start completed.
2018-10-09T09:42:00.819Z [main] ....  INFO --- DatabaseFactory: Database: jdbc:mariadb://address=(host=galera-mgmt.service.consul)(port=3306)(type=master)/credhub (MySQL 10.1)
2018-10-09T09:42:01.008Z [main] ....  INFO --- DbValidate: Successfully validated 65 migrations (execution time 00:00.114s)
2018-10-09T09:42:01.083Z [main] ....  INFO --- DbMigrate: Current version of schema `credhub`: 48
2018-10-09T09:42:01.093Z [main] ....  INFO --- DbMigrate: Migrating schema `credhub` to version 49 - add path to permission table

2018-10-09T09:42:06.728Z [main] ....  WARN --- ProxyConnection: HikariPool-1 - Connection org.mariadb.jdbc.MariaDbConnection@6304101a marked as broken because of SQLSTATE(08), ErrorCode(0)
java.sql.SQLNonTransientConnectionException: (conn=15993114) unexpected end of stream, read 0 bytes from 4 (socket was closed by server)
        at org.mariadb.jdbc.internal.util.exceptions.ExceptionMapper.get(ExceptionMapper.java:175) ~[mariadb-java-client-2.2.1.jar!/:?]
        at org.mariadb.jdbc.internal.util.exceptions.ExceptionMapper.getException(ExceptionMapper.java:110) ~[mariadb-java-client-2.2.1.jar!/:?]
        at org.mariadb.jdbc.MariaDbStatement.executeExceptionEpilogue(MariaDbStatement.java:228) ~[mariadb-java-client-2.2.1.jar!/:?]
        at org.mariadb.jdbc.MariaDbStatement.executeInternal(MariaDbStatement.java:334) ~[mariadb-java-client-2.2.1.jar!/:?]
        at org.mariadb.jdbc.MariaDbStatement.execute(MariaDbStatement.java:386) ~[mariadb-java-client-2.2.1.jar!/:?]
        at com.zaxxer.hikari.pool.ProxyStatement.execute(ProxyStatement.java:95) ~[HikariCP-2.7.9.jar!/:?]
        at com.zaxxer.hikari.pool.HikariProxyStatement.execute(HikariProxyStatement.java) ~[HikariCP-2.7.9.jar!/:?]
        at org.flywaydb.core.internal.util.jdbc.JdbcTemplate.executeStatement(JdbcTemplate.java:265) ~[flyway-core-5.0.7.jar!/:?]
        at org.flywaydb.core.internal.database.StandardSqlStatement.execute(StandardSqlStatement.java:41) ~[flyway-core-5.0.7.jar!/:?]
        at org.flywaydb.core.internal.database.ExecutableSqlScript.execute(ExecutableSqlScript.java:128) ~[flyway-core-5.0.7.jar!/:?]
        at org.flywaydb.core.internal.resolver.sql.SqlMigrationExecutor.execute(SqlMigrationExecutor.java:75) ~[flyway-core-5.0.7.jar!/:?]
        at org.flywaydb.core.internal.command.DbMigrate.doMigrateGroup(DbMigrate.java:363) ~[flyway-core-5.0.7.jar!/:?]
        at org.flywaydb.core.internal.command.DbMigrate.access$400(DbMigrate.java:52) ~[flyway-core-5.0.7.jar!/:?]
        at org.flywaydb.core.internal.command.DbMigrate$5.call(DbMigrate.java:297) ~[flyway-core-5.0.7.jar!/:?]
        at org.flywaydb.core.internal.util.jdbc.TransactionTemplate.execute(TransactionTemplate.java:75) ~[flyway-core-5.0.7.jar!/:?]
        at org.flywaydb.core.internal.command.DbMigrate.applyMigrations(DbMigrate.java:294) ~[flyway-core-5.0.7.jar!/:?]
        at org.flywaydb.core.internal.command.DbMigrate.migrateGroup(DbMigrate.java:259) ~[flyway-core-5.0.7.jar!/:?]
        at org.flywaydb.core.internal.command.DbMigrate.access$300(DbMigrate.java:52) ~[flyway-core-5.0.7.jar!/:?]
        at org.flywaydb.core.internal.command.DbMigrate$4.call(DbMigrate.java:179) ~[flyway-core-5.0.7.jar!/:?]
        at org.flywaydb.core.internal.command.DbMigrate$4.call(DbMigrate.java:176) ~[flyway-core-5.0.7.jar!/:?]
        at org.flywaydb.core.internal.database.mysql.MySQLNamedLockTemplate.execute(MySQLNamedLockTemplate.java:60) ~[flyway-core-5.0.7.jar!/:?]
        at org.flywaydb.core.internal.database.mysql.MySQLConnection.lock(MySQLConnection.java:78) ~[flyway-core-5.0.7.jar!/:?]
        at org.flywaydb.core.internal.schemahistory.JdbcTableSchemaHistory.lock(JdbcTableSchemaHistory.java:148) ~[flyway-core-5.0.7.jar!/:?]
        at org.flywaydb.core.internal.command.DbMigrate.migrateAll(DbMigrate.java:176) ~[flyway-core-5.0.7.jar!/:?]
        at org.flywaydb.core.internal.command.DbMigrate.migrate(DbMigrate.java:145) ~[flyway-core-5.0.7.jar!/:?]
        at org.flywaydb.core.Flyway$1.execute(Flyway.java:1206) ~[flyway-core-5.0.7.jar!/:?]
        at org.flywaydb.core.Flyway$1.execute(Flyway.java:1168) ~[flyway-core-5.0.7.jar!/:?]
        at org.flywaydb.core.Flyway.execute(Flyway.java:1655) ~[flyway-core-5.0.7.jar!/:?]
        at org.flywaydb.core.Flyway.migrate(Flyway.java:1168) ~[flyway-core-5.0.7.jar!/:?]
        at org.springframework.boot.autoconfigure.flyway.FlywayMigrationInitializer.afterPropertiesSet(FlywayMigrationInitializer.java:66) ~[spring-boot-autoconfigure-2.0.4.RELEASE.jar!/:2.0.4.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1758) ~[spring-beans-5.0.8.RELEASE.jar!/:5.0.8.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1695) ~[spring-beans-5.0.8.RELEASE.jar!/:5.0.8.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:573) ~[spring-beans-5.0.8.RELEASE.jar!/:5.0.8.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:495) ~[spring-beans-5.0.8.RELEASE.jar!/:5.0.8.RELEASE]
        at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:317) ~[spring-beans-5.0.8.RELEASE.jar!/:5.0.8.RELEASE]
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222) [spring-beans-5.0.8.RELEASE.jar!/:5.0.8.RELEASE]
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:315) [spring-beans-5.0.8.RELEASE.jar!/:5.0.8.RELEASE]
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199) [spring-beans-5.0.8.RELEASE.jar!/:5.0.8.RELEASE]
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:304) [spring-beans-5.0.8.RELEASE.jar!/:5.0.8.RELEASE]
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199) [spring-beans-5.0.8.RELEASE.jar!/:5.0.8.RELEASE]
        at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1089) [spring-context-5.0.8.RELEASE.jar!/:5.0.8.RELEASE]
        at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:859) [spring-context-5.0.8.RELEASE.jar!/:5.0.8.RELEASE]
        at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:550) [spring-context-5.0.8.RELEASE.jar!/:5.0.8.RELEASE]
        at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:140) [spring-boot-2.0.4.RELEASE.jar!/:2.0.4.RELEASE]
        at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:762) [spring-boot-2.0.4.RELEASE.jar!/:2.0.4.RELEASE]
        at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:398) [spring-boot-2.0.4.RELEASE.jar!/:2.0.4.RELEASE]
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:330) [spring-boot-2.0.4.RELEASE.jar!/:2.0.4.RELEASE]
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:1258) [spring-boot-2.0.4.RELEASE.jar!/:2.0.4.RELEASE]
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:1246) [spring-boot-2.0.4.RELEASE.jar!/:2.0.4.RELEASE]
        at org.cloudfoundry.credhub.CredentialManagerApp.main(CredentialManagerApp.java:29) [classes!/:?]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_181]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_181]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_181]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_181]
        at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48) [credhub.jar:?]
        at org.springframework.boot.loader.Launcher.launch(Launcher.java:87) [credhub.jar:?]
        at org.springframework.boot.loader.Launcher.launch(Launcher.java:50) [credhub.jar:?]
        at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:51) [credhub.jar:?]
Caused by: java.sql.SQLException: unexpected end of stream, read 0 bytes from 4 (socket was closed by server)
Query is: ALTER TABLE permission
  ADD COLUMN path VARCHAR(255)
        at org.mariadb.jdbc.internal.util.LogQueryTool.exceptionWithQuery(LogQueryTool.java:119) ~[mariadb-java-client-2.2.1.jar!/:?]
        at org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.executeQuery(AbstractQueryProtocol.java:198) ~[mariadb-java-client-2.2.1.jar!/:?]
        at org.mariadb.jdbc.MariaDbStatement.executeInternal(MariaDbStatement.java:328) ~[mariadb-java-client-2.2.1.jar!/:?]
        ... 54 more
Caused by: java.io.EOFException: unexpected end of stream, read 0 bytes from 4 (socket was closed by server)
        at org.mariadb.jdbc.internal.io.input.StandardPacketInputStream.getPacketArray(StandardPacketInputStream.java:239) ~[mariadb-java-client-2.2.1.jar!/:?]
        at org.mariadb.jdbc.internal.io.input.StandardPacketInputStream.getPacket(StandardPacketInputStream.java:207) ~[mariadb-java-client-2.2.1.jar!/:?]
        at org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.readPacket(AbstractQueryProtocol.java:1346) ~[mariadb-java-client-2.2.1.jar!/:?]
        at org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.getResult(AbstractQueryProtocol.java:1327) ~[mariadb-java-client-2.2.1.jar!/:?]
        at org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.executeQuery(AbstractQueryProtocol.java:195) ~[mariadb-java-client-2.2.1.jar!/:?]
        at org.mariadb.jdbc.MariaDbStatement.executeInternal(MariaDbStatement.java:328) ~[mariadb-java-client-2.2.1.jar!/:?]
        ... 54 more
2018-10-09T09:42:06.739Z [main] .... ERROR --- TransactionTemplate: Unable to rollback transaction

I searched this error with DuckDuckGo and found Migrating from MySql: MariaDB server closing client connections unexpectedly

all the recommended solution were already in place in our setup. For example:

| wait_timeout                        | 28800    |

Then it must be some Galera replication issue.

(none)> show global variables like '%version%';
+-------------------------+---------------------------+
| Variable_name           | Value                     |
+-------------------------+---------------------------+
| innodb_version          | 5.6.39-83.1               |
| protocol_version        | 10                        |
| slave_type_conversions  |                           |
| version                 | 10.1.34-MariaDB           |
| version_comment         | Source distribution       |
| version_compile_machine | x86_64                    |
| version_compile_os      | Linux                     |
| version_malloc_library  | system                    |
| version_ssl_library     | OpenSSL 1.0.1f 6 Jan 2014 |
| wsrep_patch_version     | wsrep_25.23               |
+-------------------------+---------------------------+

We use 3 node Galera cluster. With proxy we direct Apps to only one node. The "primary" has no log entries during this time.

node 1:

2018-10-09  9:42:20 140623337528064 [ERROR] Slave SQL: Error 'Duplicate column name 'path'' on query. Default database: 'credhub'. Query: 'ALTER TABLE permission
  ADD COLUMN path VARCHAR(255)', Internal MariaDB error code: 1060
2018-10-09  9:42:20 140623337528064 [Warning] WSREP: RBR event 1 Query apply warning: 1, 381109243
2018-10-09  9:42:20 140623337528064 [Warning] WSREP: Ignoring error for TO isolated action: source: 9f0f4ed6-aa9f-11e8-88d4-bb61e2d431d5 version: 3 local: 0 state: APPLYING flags: 65 conn_id: 137099 trx_id: -1 seqnos (l: 72883646, g: 381109243, s: 381109168, d: 381109242, ts: 3699202329934779)
2018-10-09 10:32:51 140623337528064 [ERROR] Slave SQL: Error 'Duplicate column name 'path'' on query. Default database: 'credhub'. Query: 'ALTER TABLE permission
  ADD COLUMN path VARCHAR(255)', Internal MariaDB error code: 1060
2018-10-09 10:32:51 140623337528064 [Warning] WSREP: RBR event 1 Query apply warning: 1, 381173176
2018-10-09 10:32:51 140623337528064 [Warning] WSREP: Ignoring error for TO isolated action: source: 2e608d33-aa9f-11e8-ad98-87453d4293df version: 3 local: 0 state: APPLYING flags: 65 conn_id: 16009494 trx_id: -1 seqnos (l: 72948512, g: 381173176, s: 381173175, d: 381173175, ts: 3702645177012081)
2018-10-09 10:34:30 140623337528064 [ERROR] Slave SQL: Error 'Duplicate column name 'path'' on query. Default database: 'credhub'. Query: 'ALTER TABLE permission
  ADD COLUMN path VARCHAR(255)', Internal MariaDB error code: 1060
2018-10-09 10:34:30 140623337528064 [Warning] WSREP: RBR event 1 Query apply warning: 1, 381175187
2018-10-09 10:34:30 140623337528064 [Warning] WSREP: Ignoring error for TO isolated action: source: 2e608d33-aa9f-11e8-ad98-87453d4293df version: 3 local: 0 state: APPLYING flags: 65 conn_id: 16009893 trx_id: -1 seqnos (l: 72950553, g: 381175187, s: 381175186, d: 381175186, ts: 3702744192862858)
2018-10-09 10:38:23 140623337528064 [ERROR] Slave SQL: Error 'Can't DROP 'path'; check that column/key exists' on query. Default database: 'credhub'. Query: 'ALTER TABLE permission
  DROP COLUMN path', Internal MariaDB error code: 1091
2018-10-09 10:38:23 140623337528064 [Warning] WSREP: RBR event 1 Query apply warning: 1, 381180154
2018-10-09 10:38:23 140623337528064 [Warning] WSREP: Ignoring error for TO isolated action: source: 2e608d33-aa9f-11e8-ad98-87453d4293df version: 3 local: 0 state: APPLYING flags: 65 conn_id: 16009214 trx_id: -1 seqnos (l: 72955600, g: 381180154, s: 381180153, d: 381180153, ts: 3702978039461852)

node 2:

2018-10-09 10:32:51 139950574152448 [ERROR] Slave SQL: Error 'Duplicate column name 'path'' on query. Default database: 'credhub'. Query: 'ALTER TABLE permission
  ADD COLUMN path VARCHAR(255)', Internal MariaDB error code: 1060
2018-10-09 10:32:51 139950574152448 [Warning] WSREP: RBR event 1 Query apply warning: 1, 381173176
2018-10-09 10:32:51 139950574152448 [Warning] WSREP: Ignoring error for TO isolated action: source: 2e608d33-aa9f-11e8-ad98-87453d4293df version: 3 local: 0 state: APPLYING flags: 65 conn_id: 16009494 trx_id: -1 seqnos (l: 72941962, g: 381173176, s: 381173175, d: 381173175, ts: 3702645177012081)
2018-10-09 10:34:30 139950574152448 [ERROR] Slave SQL: Error 'Duplicate column name 'path'' on query. Default database: 'credhub'. Query: 'ALTER TABLE permission
  ADD COLUMN path VARCHAR(255)', Internal MariaDB error code: 1060
2018-10-09 10:34:30 139950574152448 [Warning] WSREP: RBR event 1 Query apply warning: 1, 381175187
2018-10-09 10:34:30 139950574152448 [Warning] WSREP: Ignoring error for TO isolated action: source: 2e608d33-aa9f-11e8-ad98-87453d4293df version: 3 local: 0 state: APPLYING flags: 65 conn_id: 16009893 trx_id: -1 seqnos (l: 72944003, g: 381175187, s: 381175186, d: 381175186, ts: 3702744192862858)
2018-10-09 10:38:23 139950574152448 [ERROR] Slave SQL: Error 'Can't DROP 'path'; check that column/key exists' on query. Default database: 'credhub'. Query: 'ALTER TABLE permission
  DROP COLUMN path', Internal MariaDB error code: 1091
2018-10-09 10:38:23 139950574152448 [Warning] WSREP: RBR event 1 Query apply warning: 1, 381180154
2018-10-09 10:38:23 139950574152448 [Warning] WSREP: Ignoring error for TO isolated action: source: 2e608d33-aa9f-11e8-ad98-87453d4293df version: 3 local: 0 state: APPLYING flags: 65 conn_id: 16009214 trx_id: -1 seqnos (l: 72949050, g: 381180154, s: 381180153, d: 381180153, ts: 3702978039461852)

The table in question is tiny:

MariaDB [credhub]> select count(*) from permission;
+----------+
| count(*) |
+----------+
|      308 |
+----------+
1 row in set (0.00 sec)

MariaDB [credhub]> show create table permission;
+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Table      | Create Table                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| permission | CREATE TABLE `permission` (
  `uuid` binary(16) NOT NULL,
  `actor` varchar(255) NOT NULL,
  `read_permission` tinyint(1) NOT NULL DEFAULT '0',
  `write_permission` tinyint(1) NOT NULL DEFAULT '0',
  `delete_permission` tinyint(1) NOT NULL DEFAULT '0',
  `read_acl_permission` tinyint(1) NOT NULL DEFAULT '0',
  `write_acl_permission` tinyint(1) NOT NULL DEFAULT '0',
  `path` varchar(255) NOT NULL,
  PRIMARY KEY (`uuid`),
  UNIQUE KEY `permission_path_actor_uindex` (`path`,`actor`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 |
+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.01 sec)

MariaDB [credhub]> SELECT   TABLE_NAME AS `Table`,   ROUND((DATA_LENGTH + INDEX_LENGTH) / 1024 / 1024) AS `Size (MB)` FROM   information_schema.TABLES WHERE   TABLE_SCHEMA = "credhub" ORDER BY   (DATA_LENGTH + INDEX_LENGTH) DESC;
+------------------------+-----------+
| Table                  | Size (MB) |
+------------------------+-----------+
| encrypted_value        |         7 |
| credential_version     |         0 |
| credential             |         0 |
| permission             |         0 |
| certificate_credential |         0 |
| password_credential    |         0 |
| user_credential        |         0 |
| flyway_schema_history  |         0 |
| encryption_key_canary  |         0 |
| rsa_credential         |         0 |
| ssh_credential         |         0 |
+------------------------+-----------+
11 rows in set (0.01 sec)

Any idea howto fix this issue?

Im using Postgres 10, and I was wondering if its possible to cascade a logical replication from a binary slave replica? I am looking to have databases with the same data but indexed in a different way is this possible and makes sense?