On 09-05-16 - 08:41:51 Replica Set elected a new primary. The question why?
Here the logs from this time series and without login messages egrep -v 'NETWORK|ACCESS'.
On the same OpenStack platform with SDN are running lots of replica sets which didn't switch this timestamp.
We are running MongoDB 3.2.1 on RHEL7 VMs.
2016-05-09T08:40:01.632+0200 I COMMAND [conn111484] command rammyD.Objects command: getMore { getMore: 52899238286, collection: "Objects" } cursorid:52899238286 cursorExhausted:1 keyUpdates:0 writeConflicts:0 numYields:23 nreturned:2976 reslen:511975 locks:{ Global: { acquireCount: { r: 48 } }, Database: { acquireCount: { r: 24 } }, Collection: { acquireCount: { r: 24 } } } protocol:op_query 104ms
2016-05-09T08:40:02.165+0200 I COMMAND [conn111454] command rammyD.Objects command: getMore { getMore: 53404336016, collection: "Objects" } cursorid:53404336016 cursorExhausted:1 keyUpdates:0 writeConflicts:0 numYields:24 nreturned:3163 reslen:544041 locks:{ Global: { acquireCount: { r: 50 } }, Database: { acquireCount: { r: 25 } }, Collection: { acquireCount: { r: 25 } } } protocol:op_query 152ms
2016-05-09T08:40:07.652+0200 I COMMAND [conn111483] command rammyD.Objects command: getMore { getMore: 52112174666, collection: "Objects" } cursorid:52112174666 cursorExhausted:1 keyUpdates:0 writeConflicts:0 numYields:134 nreturned:17223 reslen:2983971 locks:{ Global: { acquireCount: { r: 270 } }, Database: { acquireCount: { r: 135 } }, Collection: { acquireCount: { r: 135 } } } protocol:op_query 367ms
2016-05-09T08:40:08.861+0200 I COMMAND [conn111483] command rammyD.Objects command: getMore { getMore: 53089137757, collection: "Objects" } cursorid:53089137757 cursorExhausted:1 keyUpdates:0 writeConflicts:0 numYields:21 nreturned:2588 reslen:441271 locks:{ Global: { acquireCount: { r: 44 } }, Database: { acquireCount: { r: 22 } }, Collection: { acquireCount: { r: 22 } } } protocol:op_query 246ms
2016-05-09T08:40:09.818+0200 I COMMAND [conn111397] command rammyD.Objects command: getMore { getMore: 53659787468, collection: "Objects" } cursorid:53659787468 cursorExhausted:1 keyUpdates:0 writeConflicts:0 numYields:6 nreturned:558 reslen:95291 locks:{ Global: { acquireCount: { r: 14 } }, Database: { acquireCount: { r: 7 } }, Collection: { acquireCount: { r: 7 } } } protocol:op_query 103ms
2016-05-09T08:40:10.394+0200 I COMMAND [conn111484] command rammyD.Objects command: getMore { getMore: 55354201324, collection: "Objects" } cursorid:55354201324 cursorExhausted:1 keyUpdates:0 writeConflicts:0 numYields:163 nreturned:20667 reslen:3583628 locks:{ Global: { acquireCount: { r: 328 } }, Database: { acquireCount: { r: 164 } }, Collection: { acquireCount: { r: 164 } } } protocol:op_query 1095ms
2016-05-09T08:40:10.539+0200 I COMMAND [conn111482] command rammyD.Objects command: getMore { getMore: 51978741855, collection: "Objects" } cursorid:51978741855 keyUpdates:0 writeConflicts:0 numYields:104 nreturned:12811 reslen:4194510 locks:{ Global: { acquireCount: { r: 210 } }, Database: { acquireCount: { r: 105 } }, Collection: { acquireCount: { r: 105 } } } protocol:op_query 908ms
2016-05-09T08:40:11.344+0200 I COMMAND [conn111482] command rammyD.Objects command: getMore { getMore: 51978741855, collection: "Objects" } cursorid:51978741855 keyUpdates:0 writeConflicts:0 numYields:98 nreturned:12580 reslen:4194477 locks:{ Global: { acquireCount: { r: 198 } }, Database: { acquireCount: { r: 99 } }, Collection: { acquireCount: { r: 99 } } } protocol:op_query 456ms
2016-05-09T08:40:12.056+0200 I COMMAND [conn111482] command rammyD.Objects command: getMore { getMore: 51978741855, collection: "Objects" } cursorid:51978741855 cursorExhausted:1 keyUpdates:0 writeConflicts:0 numYields:76 nreturned:9721 reslen:3141047 locks:{ Global: { acquireCount: { r: 154 } }, Database: { acquireCount: { r: 77 } }, Collection: { acquireCount: { r: 77 } } } protocol:op_query 380ms
2016-05-09T08:40:22.221+0200 I COMMAND [conn111479] command rammyD.Objects command: getMore { getMore: 54413844664, collection: "Objects" } cursorid:54413844664 cursorExhausted:1 keyUpdates:0 writeConflicts:0 numYields:23 nreturned:2970 reslen:508733 locks:{ Global: { acquireCount: { r: 48 } }, Database: { acquireCount: { r: 24 } }, Collection: { acquireCount: { r: 24 } } } protocol:op_query 103ms
2016-05-09T08:40:27.964+0200 I COMMAND [conn111546] command rammyD.Objects command: getMore { getMore: 54136275592, collection: "Objects" } cursorid:54136275592 cursorExhausted:1 keyUpdates:0 writeConflicts:0 numYields:36 nreturned:4639 reslen:800822 locks:{ Global: { acquireCount: { r: 74 } }, Database: { acquireCount: { r: 37 } }, Collection: { acquireCount: { r: 37 } } } protocol:op_query 151ms
2016-05-09T08:41:39.004+0200 I COMMAND [conn48759] command local.oplog.rs command: getMore { getMore: 29159826825, collection: "oplog.rs", maxTimeMS: 2000 } cursorid:29159826825 keyUpdates:0 writeConflicts:0 numYields:1 nreturned:0 reslen:86 locks:{ Global: { acquireCount: { r: 6 } }, Database: { acquireCount: { r: 3 } }, oplog: { acquireCount: { r: 3 } } } protocol:op_command 69432ms
2016-05-09T08:41:39.005+0200 I COMMAND [conn110443] command local.oplog.rs command: getMore { getMore: 29201452228, collection: "oplog.rs", maxTimeMS: 2000 } cursorid:29201452228 keyUpdates:0 writeConflicts:0 numYields:1 nreturned:0 reslen:86 locks:{ Global: { acquireCount: { r: 6 } }, Database: { acquireCount: { r: 3 } }, oplog: { acquireCount: { r: 3 } } } protocol:op_command 69433ms
2016-05-09T08:41:39.080+0200 I COMMAND [conn23747] command admin.$cmd command: isMaster { isMaster: 1 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:824 locks:{} protocol:op_query 71488ms
2016-05-09T08:41:39.080+0200 I COMMAND [conn45085] command admin.$cmd command: isMaster { isMaster: 1 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:824 locks:{} protocol:op_query 71487ms
2016-05-09T08:41:39.081+0200 I COMMAND [conn47432] command admin.$cmd command: isMaster { isMaster: 1 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:824 locks:{} protocol:op_query 71485ms
2016-05-09T08:41:39.085+0200 I COMMAND [conn49893] command admin.$cmd command: isMaster { isMaster: 1 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:824 locks:{} protocol:op_query 71366ms
2016-05-09T08:41:39.088+0200 I COMMAND [conn49462] command admin.$cmd command: isMaster { isMaster: 1 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:824 locks:{} protocol:op_query 71355ms
2016-05-09T08:41:39.088+0200 I COMMAND [conn57769] command admin.$cmd command: isMaster { isMaster: 1 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:824 locks:{} protocol:op_query 70521ms
2016-05-09T08:41:39.088+0200 I COMMAND [conn111532] command admin.$cmd command: isMaster { ismaster: 1 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:824 locks:{} protocol:op_query 70477ms
2016-05-09T08:41:39.089+0200 I COMMAND [conn57323] command admin.$cmd command: isMaster { isMaster: 1 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:824 locks:{} protocol:op_query 71206ms
2016-05-09T08:41:39.089+0200 I COMMAND [conn49757] command admin.$cmd command: isMaster { isMaster: 1 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:824 locks:{} protocol:op_query 71186ms
2016-05-09T08:41:39.090+0200 I COMMAND [conn55358] command admin.$cmd command: isMaster { isMaster: 1 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:824 locks:{} protocol:op_query 70924ms
2016-05-09T08:41:39.091+0200 I COMMAND [conn52196] command admin.$cmd command: isMaster { isMaster: 1 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:824 locks:{} protocol:op_query 70816ms
2016-05-09T08:41:39.091+0200 I COMMAND [conn57326] command admin.$cmd command: isMaster { isMaster: 1 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:824 locks:{} protocol:op_query 70767ms
2016-05-09T08:41:39.091+0200 I COMMAND [conn57876] command admin.$cmd command: isMaster { isMaster: 1 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:824 locks:{} protocol:op_query 70666ms
2016-05-09T08:41:39.091+0200 I COMMAND [conn55361] command admin.$cmd command: isMaster { isMaster: 1 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:824 locks:{} protocol:op_query 70663ms
2016-05-09T08:41:39.095+0200 I COMMAND [conn58457] command admin.$cmd command: isMaster { isMaster: 1 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:824 locks:{} protocol:op_query 70497ms
2016-05-09T08:41:39.095+0200 I COMMAND [conn57344] command admin.$cmd command: isMaster { isMaster: 1 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:824 locks:{} protocol:op_query 70360ms
2016-05-09T08:41:39.095+0200 I COMMAND [conn12675] command admin.$cmd command: isMaster { isMaster: 1 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:824 locks:{} protocol:op_query 70222ms
2016-05-09T08:41:39.096+0200 I COMMAND [conn58181] command admin.$cmd command: isMaster { isMaster: 1 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:824 locks:{} protocol:op_query 70214ms
2016-05-09T08:41:39.096+0200 I COMMAND [conn58261] command admin.$cmd command: isMaster { isMaster: 1 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:824 locks:{} protocol:op_query 70135ms
2016-05-09T08:41:39.105+0200 I COMMAND [conn111543] command admin.$cmd command: isMaster { ismaster: 1 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:824 locks:{} protocol:op_query 71529ms
2016-05-09T08:41:39.108+0200 I COMMAND [conn54972] command admin.$cmd command: isMaster { isMaster: 1 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:824 locks:{} protocol:op_query 70100ms
2016-05-09T08:41:39.108+0200 I COMMAND [conn111524] command admin.$cmd command: isMaster { isMaster: 1 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:824 locks:{} protocol:op_query 69911ms
2016-05-09T08:41:39.113+0200 I COMMAND [conn57352] command admin.$cmd command: isMaster { isMaster: 1 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:824 locks:{} protocol:op_query 69758ms
2016-05-09T08:41:39.117+0200 I COMMAND [conn12676] command admin.$cmd command: isMaster { isMaster: 1 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:824 locks:{} protocol:op_query 69749ms
2016-05-09T08:41:39.117+0200 I COMMAND [conn57874] command admin.$cmd command: isMaster { isMaster: 1 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:824 locks:{} protocol:op_query 69724ms
2016-05-09T08:41:39.117+0200 I COMMAND [conn23834] command admin.$cmd command: isMaster { isMaster: 1 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:824 locks:{} protocol:op_query 69640ms
2016-05-09T08:41:39.118+0200 I COMMAND [conn110092] command admin.$cmd command: isMaster { isMaster: 1 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:824 locks:{} protocol:op_query 69573ms
2016-05-09T08:41:39.118+0200 I COMMAND [conn22813] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "sffre-replicaset", pv: 1, v: 19673, from: "zwdfhf5fu-ter-2.zabbix.lu:27000", fromId: 3, checkEmpty: false } keyUpdates:0 writeConflicts:0 numYields:0 reslen:325 locks:{} protocol:op_command 70994ms
2016-05-09T08:41:39.118+0200 I COMMAND [conn110411] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "sffre-replicaset", pv: 1, v: 19673, from: "zwdfhf5fu-ter-4.zabbix.lu:27000", fromId: 4, checkEmpty: false } keyUpdates:0 writeConflicts:0 numYields:0 reslen:325 locks:{} protocol:op_command 70363ms
2016-05-09T08:41:39.118+0200 I COMMAND [conn22809] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "sffre-replicaset", pv: 1, v: 19673, from: "zwdfhf5fu-ter-5.zabbix.lu:27000", fromId: 5, checkEmpty: false } keyUpdates:0 writeConflicts:0 numYields:0 reslen:325 locks:{} protocol:op_command 70185ms
2016-05-09T08:41:39.119+0200 I COMMAND [conn22805] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "sffre-replicaset", pv: 1, v: 19673, from: "zwdfhf5fu-ter-3.zabbix.lu:27000", fromId: 2, checkEmpty: false } keyUpdates:0 writeConflicts:0 numYields:0 reslen:325 locks:{} protocol:op_command 70152ms
2016-05-09T08:41:39.147+0200 I COMMAND [conn111529] command rammyD.Objects command: find { find: "Objects", filter: { Container: ObjectId('570523074644a406fa1fff25'), Attributes: 1, Type: 1, Domain: 2 }, sort: { SourceTakenTime: -1 }, projection: { ETag: 1, Extension: 1, Mime: 1, SourceTakenTime: 1, Length: 1, SystemCreationTime: 1, SystemDeletionTime: 1 } } planSummary: IXSCAN { Container: 1, Attributes: 1, Type: 1, Domain: 1, SourceTakenTime: 1 } cursorid:54675343067 keysExamined:101 docsExamined:101 keyUpdates:0 writeConflicts:0 numYields:4 nreturned:101 reslen:17389 locks:{ Global: { acquireCount: { r: 10 } }, Database: { acquireCount: { r: 5 } }, Collection: { acquireCount: { r: 5 } } } protocol:op_query 116ms
2016-05-09T08:41:39.147+0200 I COMMAND [conn111442] command rammyD.Objects command: find { find: "Objects", filter: { Container: ObjectId('5655bf192af7bf001e46349a'), Attributes: 1, Type: 1, Domain: 2 }, sort: { SourceTakenTime: -1 }, projection: { ETag: 1, Extension: 1, Mime: 1, SourceTakenTime: 1, Length: 1, SystemCreationTime: 1, SystemDeletionTime: 1 } } planSummary: IXSCAN { Container: 1, Attributes: 1, Type: 1, Domain: 1, SourceTakenTime: 1 } cursorid:54087089791 keysExamined:101 docsExamined:101 keyUpdates:0 writeConflicts:0 numYields:3 nreturned:101 reslen:17262 locks:{ Global: { acquireCount: { r: 8 } }, Database: { acquireCount: { r: 4 } }, Collection: { acquireCount: { r: 4 } } } protocol:op_query 116ms
2016-05-09T08:41:39.151+0200 I COMMAND [conn111556] command rammyD.Objects command: find { find: "Objects", filter: { Container: ObjectId('570baa5f7e66254dfe93f664'), Attributes: 1, Type: 1, Domain: 2 }, sort: { SourceTakenTime: -1 }, projection: { ETag: 1, Extension: 1, Mime: 1, SourceTakenTime: 1, Length: 1, SystemCreationTime: 1, SystemDeletionTime: 1 } } planSummary: IXSCAN { Container: 1, Attributes: 1, Type: 1, Domain: 1, SourceTakenTime: 1 } cursorid:54828351582 keysExamined:101 docsExamined:101 keyUpdates:0 writeConflicts:0 numYields:3 nreturned:101 reslen:17147 locks:{ Global: { acquireCount: { r: 8 } }, Database: { acquireCount: { r: 4 } }, Collection: { acquireCount: { r: 4 } } } protocol:op_query 134ms
2016-05-09T08:41:39.151+0200 I REPL [ReplicationExecutor] Member zwdfhf5fu-ter-3.zabbix.lu:27000 is now in state PRIMARY
2016-05-09T08:41:39.151+0200 I REPL [ReplicationExecutor] another primary seen with election time 573030e6:1 my election time is 57240978:1
2016-05-09T08:41:39.151+0200 I REPL [ReplicationExecutor] stepping down; another primary was elected more recently
2016-05-09T08:41:39.151+0200 I REPL [ReplicationExecutor] Stepping down from primary in response to heartbeat
2016-05-09T08:41:39.151+0200 I COMMAND [conn111508] command rammyD.Objects command: find { find: "Objects", filter: { Container: ObjectId('5727af23e08a5b0716cae10f'), Attributes: 1, Type: 1, Domain: 2 }, sort: { SourceTakenTime: -1 }, projection: { ETag: 1, Extension: 1, Mime: 1, SourceTakenTime: 1, Length: 1, SystemCreationTime: 1, SystemDeletionTime: 1 } } planSummary: IXSCAN { Container: 1, Attributes: 1, Type: 1, Domain: 1, SourceTakenTime: 1 } cursorid:54479367894 keysExamined:101 docsExamined:101 keyUpdates:0 writeConflicts:0 numYields:4 nreturned:101 reslen:17385 locks:{ Global: { acquireCount: { r: 10 } }, Database: { acquireCount: { r: 5 } }, Collection: { acquireCount: { r: 5 } } } protocol:op_query 131ms
2016-05-09T08:41:39.155+0200 I REPL [ReplicationExecutor] another primary seen with election time 573030e6:1 my election time is 57240978:1
2016-05-09T08:41:39.155+0200 I REPL [ReplicationExecutor] another primary seen with election time 573030e6:1 my election time is 57240978:1
2016-05-09T08:41:39.158+0200 I COMMAND [conn111406] command rammyD.Objects command: find { find: "Objects", filter: { Container: ObjectId('570de3acf97e84d818a1a7b8'), Attributes: 1, Type: 1, Domain: 2 }, sort: { SourceTakenTime: -1 }, projection: { ETag: 1, Extension: 1, Mime: 1, SourceTakenTime: 1, Length: 1, SystemCreationTime: 1, SystemDeletionTime: 1 } } planSummary: IXSCAN { Container: 1, Attributes: 1, Type: 1, Domain: 1, SourceTakenTime: 1 } cursorid:51833266944 keysExamined:101 docsExamined:101 keyUpdates:0 writeConflicts:0 numYields:3 nreturned:101 reslen:17396 locks:{ Global: { acquireCount: { r: 8 } }, Database: { acquireCount: { r: 4 } }, Collection: { acquireCount: { r: 4 } } } protocol:op_query 128ms
2016-05-09T08:41:39.159+0200 I COMMAND [ftdc] serverStatus was very slow: { after basic: 0, after asserts: 0, after connections: 0, after extra_info: 1627, after globalLock: 1627, after locks: 1627, after network: 1627, after opcounters: 1627, after opcountersRepl: 1627, after repl: 1707, after storageEngine: 1707, after tcmalloc: 1733, after wiredTiger: 1733, at end: 1739 }
2016-05-09T08:41:39.202+0200 I COMMAND [conn22843] command local.oplog.rs command: getMore { getMore: 26679788649, collection: "oplog.rs", maxTimeMS: 2000 } cursorid:26679788649 keyUpdates:0 writeConflicts:0 numYields:0 nreturned:0 reslen:86 locks:{ Global: { acquireCount: { r: 4 } }, Database: { acquireCount: { r: 2 } }, oplog: { acquireCount: { r: 2 } } } protocol:op_command 72951ms
2016-05-09T08:41:39.205+0200 I COMMAND [conn22858] command local.oplog.rs command: getMore { getMore: 29374615863, collection: "oplog.rs", maxTimeMS: 2000 } cursorid:29374615863 keyUpdates:0 writeConflicts:0 numYields:0 nreturned:0 reslen:86 locks:{ Global: { acquireCount: { r: 4 } }, Database: { acquireCount: { r: 2 } }, oplog: { acquireCount: { r: 2 } } } protocol:op_command 72954ms
2016-05-09T08:41:39.205+0200 I REPL [replExecDBWorker-0] transition to SECONDARY
2016-05-09T08:41:39.216+0200 I COMMAND [conn111505] command rammyD.Objects command: find { find: "Objects", filter: { Container: ObjectId('572b5c97e08a5b0716cae5cc'), Attributes: 1, Type: 1, Domain: 2 }, sort: { SourceTakenTime: -1 }, projection: { ETag: 1, Extension: 1, Mime: 1, SourceTakenTime: 1, Length: 1, SystemCreationTime: 1, SystemDeletionTime: 1 } } planSummary: IXSCAN { Container: 1, Attributes: 1, Type: 1, Domain: 1, SourceTakenTime: 1 } cursorid:52404433713 keysExamined:101 docsExamined:101 keyUpdates:0 writeConflicts:0 numYields:4 nreturned:101 reslen:17389 locks:{ Global: { acquireCount: { r: 10 }, acquireWaitCount: { r: 1 }, timeAcquiringMicros: { r: 53987 } }, Database: { acquireCount: { r: 5 } }, Collection: { acquireCount: { r: 5 } } } protocol:op_query 196ms
2016-05-09T08:41:39.216+0200 I COMMAND [conn111378] command rammyD.Objects command: find { find: "Objects", filter: { Container: ObjectId('56001017b3c6f7001e1efa66'), $or: [ { Attributes: 2 } ], Type: 1, Domain: 2, ETag: { $nin: [] } }, sort: { SourceTakenTime: -1 }, projection: { ETag: 1, Extension: 1, Mime: 1, SourceTakenTime: 1, Length: 1, SystemCreationTime: 1, SystemDeletionTime: 1 } } planSummary: IXSCAN { Container: 1, Attributes: 1, Type: 1, Domain: 1, SourceTakenTime: 1 } cursorid:52982521684 keysExamined:101 docsExamined:101 keyUpdates:0 writeConflicts:0 numYields:2 nreturned:101 reslen:20207 locks:{ Global: { acquireCount: { r: 6 }, acquireWaitCount: { r: 1 }, timeAcquiringMicros: { r: 56430 } }, Database: { acquireCount: { r: 3 } }, Collection: { acquireCount: { r: 3 } } } protocol:op_query 125ms
2016-05-09T08:41:39.217+0200 I COMMAND [conn111485] command rammyD.Objects command: find { find: "Objects", filter: { Container: ObjectId('5713a0c8b20acbcbf9a1b858'), Attributes: 1, Type: 1, Domain: 2 }, sort: { SourceTakenTime: -1 }, projection: { ETag: 1, Extension: 1, Mime: 1, SourceTakenTime: 1, Length: 1, SystemCreationTime: 1, SystemDeletionTime: 1 } } planSummary: IXSCAN { Container: 1, Attributes: 1, Type: 1, Domain: 1, SourceTakenTime: 1 } cursorid:53551307751 keysExamined:101 docsExamined:101 keyUpdates:0 writeConflicts:0 numYields:3 nreturned:101 reslen:17314 locks:{ Global: { acquireCount: { r: 8 }, acquireWaitCount: { r: 1 }, timeAcquiringMicros: { r: 57732 } }, Database: { acquireCount: { r: 4 } }, Collection: { acquireCount: { r: 4 } } } protocol:op_query 136ms
2016-05-09T08:41:40.042+0200 I REPL [ReplicationExecutor] syncing from: zwdfhf5fu-ter-5.zabbix.lu:27000
2016-05-09T08:42:18.546+0200 I REPL [SyncSourceFeedback] setting syncSourceFeedback to zwdfhf5fu-ter-5.zabbix.lu:27000
2016-05-09T08:42:18.571+0200 I ASIO [NetworkInterfaceASIO-BGSync-0] Successfully connected to zwdfhf5fu-ter-5.zabbix.lu:27000
2016-05-09T08:42:31.691+0200 I COMMAND [conn111644] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "sffre-replicaset", pv: 1, v: 19673, from: "zwdfhf5fu-ter-4.zabbix.lu:27000", fromId: 4, checkEmpty: false } keyUpdates:0 writeConflicts:0 numYields:0 reslen:471 locks:{} protocol:op_command 857ms
2016-05-09T08:42:44.330+0200 I COMMAND [conn111640] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "sffre-replicaset", pv: 1, v: 19673, from: "zwdfhf5fu-ter-5.zabbix.lu:27000", fromId: 5, checkEmpty: false } keyUpdates:0 writeConflicts:0 numYields:0 reslen:471 locks:{} protocol:op_command 13496ms
2016-05-09T08:42:44.360+0200 I COMMAND [conn111642] command local.$cmd command: saslStart { saslStart: 1, mechanism: "SCRAM-SHA-1", payload: "xxx" } keyUpdates:0 writeConflicts:0 numYields:0 reslen:155 locks:{} protocol:op_query 25842ms
2016-05-09T08:42:44.379+0200 I COMMAND [conn111623] command rammyD.$cmd command: saslStart { saslStart: 1, mechanism: "SCRAM-SHA-1", payload: "xxx" } keyUpdates:0 writeConflicts:0 numYields:0 reslen:167 locks:{} protocol:op_query 25874ms
2016-05-09T08:42:44.381+0200 I COMMAND [conn111619] command rammyD.$cmd command: saslStart { saslStart: 1, mechanism: "SCRAM-SHA-1", payload: "xxx" } keyUpdates:0 writeConflicts:0 numYields:0 reslen:167 locks:{} protocol:op_query 25826ms
2016-05-09T08:42:44.382+0200 I REPL [ReplicationExecutor] Error in heartbeat request to zwdfhf5fu-ter-3.zabbix.lu:27000; ExceededTimeLimit Operation timed out
2016-05-09T08:42:44.382+0200 I REPL [ReplicationExecutor] Error in heartbeat request to zwdfhf5fu-ter-5.zabbix.lu:27000; ExceededTimeLimit Operation timed out
2016-05-09T08:42:44.382+0200 I REPL [ReplicationExecutor] Error in heartbeat request to zwdfhf5fu-ter-2.zabbix.lu:27000; ExceededTimeLimit Operation timed out
2016-05-09T08:42:44.382+0200 I REPL [ReplicationExecutor] Error in heartbeat request to zwdfhf5fu-ter-4.zabbix.lu:27000; ExceededTimeLimit Operation timed out
2016-05-09T08:42:44.409+0200 I COMMAND [conn111638] command rammyD.$cmd command: saslStart { saslStart: 1, mechanism: "SCRAM-SHA-1", payload: "xxx" } keyUpdates:0 writeConflicts:0 numYields:0 reslen:167 locks:{} protocol:op_query 25881ms
2016-05-09T08:42:44.426+0200 I COMMAND [conn111634] serverStatus was very slow: { after basic: 0, after asserts: 0, after connections: 0, after extra_info: 0, after globalLock: 0, after network: 0, after opcounters: 0, after opcountersRepl: 0, after repl: 0, after storageEngine: 0, after tcmalloc: 0, after wiredTiger: 0, at end: 1050 }
2016-05-09T08:42:44.426+0200 I COMMAND [conn111634] command admin.$cmd command: serverStatus { serverStatus: 1, locks: false, recordStats: false } keyUpdates:0 writeConflicts:0 numYields:0 reslen:18579 locks:{} protocol:op_query 25879ms
2016-05-09T08:42:44.431+0200 I COMMAND [conn111646] serverStatus was very slow: { after basic: 0, after asserts: 0, after connections: 0, after extra_info: 1024, after globalLock: 1024, after locks: 1024, after network: 1024, after opcounters: 1024, after opcountersRepl: 1024, after oplog: 1024, after repl: 1024, after storageEngine: 1024, after tcmalloc: 1024, after wiredTiger: 1024, at end: 1041 }
2016-05-09T08:42:44.431+0200 I COMMAND [conn111646] command local.oplog.rs command: serverStatus { serverStatus: 1, oplog: 1 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:19195 locks:{ Global: { acquireCount: { r: 2 } }, Database: { acquireCount: { r: 1 } }, oplog: { acquireCount: { r: 1 } } } protocol:op_query 25869ms
2016-05-09T08:42:44.602+0200 I REPL [rsBackgroundSync] starting rollback: OplogStartMissing our last op time fetched: (term: -1, timestamp: May 9 08:41:39:3). source's GTE: (term: -1, timestamp: May 9 08:41:39:3) hashes: (-3250105722725077952/-5277571331044083851)
2016-05-09T08:42:44.614+0200 I REPL [rsBackgroundSync] beginning rollback
2016-05-09T08:42:44.615+0200 I REPL [rsBackgroundSync] rollback 0
2016-05-09T08:42:44.615+0200 I REPL [ReplicationExecutor] transition to ROLLBACK
2016-05-09T08:42:44.624+0200 I REPL [rsBackgroundSync] rollback 1
2016-05-09T08:42:44.725+0200 I REPL [rsBackgroundSync] rollback 2 FindCommonPoint
2016-05-09T08:42:44.732+0200 I REPL [rsBackgroundSync] rollback our last optime: May 9 08:41:39:3
2016-05-09T08:42:44.732+0200 I REPL [rsBackgroundSync] rollback their last optime: May 9 08:42:43:1
2016-05-09T08:42:44.732+0200 I REPL [rsBackgroundSync] rollback diff in end of log times: -64 seconds
2016-05-09T08:42:45.682+0200 I REPL [rsBackgroundSync] rollback 3 fixup
2016-05-09T08:42:45.686+0200 I REPL [rsBackgroundSync] rollback 3.5
2016-05-09T08:42:45.686+0200 I REPL [rsBackgroundSync] rollback 4 n:1
2016-05-09T08:42:45.686+0200 I REPL [rsBackgroundSync] minvalid=(term: -1, timestamp: May 9 08:42:43:1)
2016-05-09T08:42:45.687+0200 I REPL [rsBackgroundSync] rollback 4.6
2016-05-09T08:42:45.687+0200 I REPL [rsBackgroundSync] rollback 4.7
2016-05-09T08:42:45.699+0200 I REPL [rsBackgroundSync] rollback 5 d:0 u:3
2016-05-09T08:42:45.699+0200 I REPL [rsBackgroundSync] rollback 6
2016-05-09T08:42:45.700+0200 I REPL [rsBackgroundSync] rollback done
2016-05-09T08:42:45.700+0200 I REPL [ReplicationExecutor] transition to RECOVERING
2016-05-09T08:42:45.700+0200 I REPL [rsBackgroundSync] rollback finished
2016-05-09T08:42:45.700+0200 I REPL [SyncSourceFeedback] SyncSourceFeedback error sending update: socket exception [SEND_ERROR] for 192.168.9.160:27000
2016-05-09T08:42:45.700+0200 I REPL [SyncSourceFeedback] updateUpstream failed: Location9001 socket exception [SEND_ERROR] for 192.168.9.160:27000, will retry
2016-05-09T08:42:45.700+0200 I REPL [ReplicationExecutor] could not find member to sync from
2016-05-09T08:42:45.758+0200 I ASIO [NetworkInterfaceASIO-Replication-0] Successfully connected to zwdfhf5fu-ter-3.zabbix.lu:27000
2016-05-09T08:42:45.773+0200 I ASIO [NetworkInterfaceASIO-Replication-0] Successfully connected to zwdfhf5fu-ter-2.zabbix.lu:27000
2016-05-09T08:42:45.773+0200 I REPL [ReplicationExecutor] Member zwdfhf5fu-ter-3.zabbix.lu:27000 is now in state PRIMARY
2016-05-09T08:42:45.773+0200 I ASIO [NetworkInterfaceASIO-Replication-0] Successfully connected to zwdfhf5fu-ter-4.zabbix.lu:27000
2016-05-09T08:42:45.788+0200 I REPL [ReplicationExecutor] Member zwdfhf5fu-ter-2.zabbix.lu:27000 is now in state SECONDARY
2016-05-09T08:42:45.788+0200 I REPL [ReplicationExecutor] Member zwdfhf5fu-ter-4.zabbix.lu:27000 is now in state SECONDARY
2016-05-09T08:42:45.788+0200 I ASIO [NetworkInterfaceASIO-Replication-0] Successfully connected to zwdfhf5fu-ter-5.zabbix.lu:27000
2016-05-09T08:42:45.789+0200 I REPL [ReplicationExecutor] Member zwdfhf5fu-ter-5.zabbix.lu:27000 is now in state SECONDARY
2016-05-09T08:42:46.332+0200 I REPL [ReplicationExecutor] transition to SECONDARY
2016-05-09T08:42:46.705+0200 I REPL [ReplicationExecutor] syncing from: zwdfhf5fu-ter-5.zabbix.lu:27000
2016-05-09T08:42:46.724+0200 I REPL [SyncSourceFeedback] setting syncSourceFeedback to zwdfhf5fu-ter-5.zabbix.lu:27000
2016-05-09T08:42:57.261+0200 I COMMAND [conn111754] killcursors: found 0 of 1
2016-05-09T08:43:02.648+0200 I ASIO [NetworkInterfaceASIO-BGSync-0] Successfully connected to zwdfhf5fu-ter-5.zabbix.lu:27000