We just went away from SQL Serer clusters in SQL Server 2008 to SQL Server 2012 with AG – have five servers with replicas in a secondary datacenter and syncing over a dedicated eline/LAN layer 2 connection - switch to switch.
We had an incident that for the duration of about 2 hours we had several logs showing that replication was terminated and then reestablished with the termination lasting about 10-20 seconds at a pop…happened “a ton” of times in those two hours and our users were complaining of latency and problems accessing the databases on the AG. This happened only on 2 of our 5 AGs that are going between the 2 datacenters.
Assuming intermittent network issues, took a look at switches, firewalls, (all redundant), the eline between and found no indications of any loss of connection issues. No indicators that the SANS had issues either. During those 2 or so hours we also got “blocking” alerts from the DB warden database that runs on all the AGs …for the two that had the replication issues.
If I have ruled out network connectivity issues, should I be looking at the SQL transactions that were running at the time? Is it possible that something in our code/long running job/query could have caused long waits or something that would have triggered the replication errors? If we have replication issues, does that cause latency on the residing databases in the AG?
I am NOT a dba, unless you count me as accidentally forced into it by default – I'm the network and sys admin/datacenter manager and SQL Server has just fallen into my realm of responsibility so this one has me grasping for explanation to our users.... any suggestions or insight?