we are running a rather highly loaded MySQL Application replicating between Master and two Slave instances. We are using Statement-based replication.
Recently, we have upgraded our hardware and a very strange problem appeared.
On Master (and slave(s)), there is an after_insert and an after_update trigger defined on Table A. Every time the trigger is fired it inserts a "log" entry into a separate table. Below is the Trigger Code:
DELIMITER |
CREATE TRIGGER orders_after_insert AFTER INSERT ON orders
FOR EACH ROW
BEGIN
DECLARE EXIT HANDLER FOR SQLEXCEPTION
RESIGNAL;
DECLARE EXIT HANDLER FOR SQLWARNING
RESIGNAL;
DECLARE EXIT HANDLER FOR NOT FOUND
RESIGNAL;
IF NEW.company_id = 1 THEN
INSERT INTO changelog SET
company_id = NEW.company_id,
dbtable = 'orders',
row_id = NEW.order_id,
created = NOW(),
action = 'INSERT';
END IF;
END;
|
DELIMITER ;
After our hardware upgrade we now face the following problem: every now and then, the trigger will NOT fire on Master, it fires okay on both slaves. As a result we have inconsistent data between Master and Slave.
- All of this is always happening inside a transaction!
- The changelog table the trigger is inserting into does NOT have a primary key. Adding an auto-increment column to the table is not an option, as it makes the trigger unsafe for replication (MySQL Warning)
- Both tables are InnoDB Tables
I am already out of ideas how to further debug this problem. It is reproducible on our production system only. Happens randomly and rarely. How can this be analyzed? How can I dig to the root of the problem?
I will be really grateful for any inputs!