I'm wondering if it is possible (within reason) to sync a new MongoDB Replica Set(RS) with an RS currently in production.
The Problem
We have a site running on AWS that has its own MongoDB Replica Set and we're working on migrating that site to Google Cloud. We want to start a separate RS on Google Compute that will catch up with the AWS RS and stay up to date during our QA lifecycle until we're ready to cut DNS over to Google.
We originally tried adding the replicas on Google as secondaries to the AWS RS to keep them in sync. Our plan was to then take the AWS RS offline and let the Google RS reassign a Primary and continue as the only RS after the site was completely migrated.
This caused some issues when one of the Google replicas got elected as a master and created some syncing issues with our current live site.
The Solution??
I've been debating if it's possible to rsync from the AWS Primary to the GCloud Primary. Is this doable with Mongo? Will the GCloud primary then properly replicate to its slaves?
Or...
Is there a better solution that anyone knows of for this problem?