We have a process which uses tableDiff. From one SQL Server 2008 R2 instance to another identical instance. It has worked very well for a long time! I have a situation where one table now has 10767594 records (and growing). So this table has been separated into its own job, its taking 2.5 hours to complete. How can I improve this?
The process is triggered by a Windows Scheduled Task which calls a .bat file.
My objective is to run this job every hour. Is there a ways to use table diff more effectively. EG if I capture the lastProcessedID can I run tableDiff next time for all records where id > lastProcessedID? Or is tableDiff just not the correct approach given the size of table now?
Please note I am not a DBA, I am an ASP developer (be gentle...).
As I said my objective is to run this job every hour. The source table from instance A can simply replace the data in the destination table/instance. I just need to update or replace one table in the destination with a table from the source. Any advice would be great? Thank you in advance
MY SOLUTION - This was a very very big surprise. The 10 million+ record table was identical on the source and destination except for 2 indexes (on the source). I applied the indexes to the source. Now I run the tableDiff job which has not been changed at all and it completes in under 2 minutes. 2.5 hours to 2 mins!
I guess tableDiff may not be a great long term solution. Will I need to implement some kind of incremental difference replication as suggested below since this table will continue to grow. At what point do you think tableDiff should be no longer used. A time factor, number of records ?