Hi I'm working in big data application, which uses a mongodb as primary and 9 secondaries. We have noticed that in the past weeks a few hosts has been out of sync, and after seeing several log messages as the following
2016-09-01T21:59:39.719+0000 I REPL [ReplicationExecutor] changing sync target because current sync target's most recent OpTime is Sep 1 21:59:07 57c8a4ab:2 which is more than 30 seconds behind member myhost:27016 whose most recent OpTime is Sep 1 21:59:39 57c8a4cb:1
The next day all of our queries where using a wrong index. This leads us to be needed to remove the secondary host, rebuild the indexes, and add it back.
Any suggestion why this could happen?