I build a replication set with one primary, one secondary and one arbiter on MongoDB 3.0.2. The primary and arbiter are on the same host and the secondary is on another host.
With the growing of write overload, the secondary can't follow the primary and step into the state of recovering. The primary can connect to the secondary as I can log to the secondary server by Mongo shell on the host of primary.
I stop all the operations and watch the secondary's state with the command rs.status() and type the command rs.syncFrom("primary's ip:port") on secondary.
Then the result of the rs.status() command shows that the optimeDate of secondary is far behind that of the primary and one message appears intermittently as below:
"set" : "shard01",
"date" : ISODate("2015-05-15T02:10:55.382Z"),
"myState" : 3,
"members" : [
{
"_id" : 0,
"name" : "xxx.xxx.xxx.xxx:xxx",
"health" : 1,
"state" : 1,
"stateStr" : "PRIMARY",
"uptime" : 135364,
"optime" : Timestamp(1431655856, 6),
"optimeDate" : ISODate("2015-05-15T02:10:56Z"),
"lastHeartbeat" : ISODate("2015-05-15T02:10:54.306Z"),
"lastHeartbeatRecv" : ISODate("2015-05-15T02:10:53.634Z"),
"pingMs" : 0,
"electionTime" : Timestamp(1431520398, 2),
"electionDate" : ISODate("2015-05-13T12:33:18Z"),
"configVersion" : 3
},
{
"_id" : 1,
"name" : "xxx.xxx.xxx.xxx:xxx",
"health" : 1,
"state" : 7,
"stateStr" : "ARBITER",
"uptime" : 135364,
"lastHeartbeat" : ISODate("2015-05-15T02:10:53.919Z"),
"lastHeartbeatRecv" : ISODate("2015-05-15T02:10:54.076Z"),
"pingMs" : 0,
"configVersion" : 3
},
{
"_id" : 2,
"name" : "xxx.xxx.xxx.xxx:xxx",
"health" : 1,
"state" : 3,
"stateStr" : "RECOVERING",
"uptime" : 135510,
"optime" : Timestamp(1431602631, 134),
"optimeDate" : ISODate("2015-05-14T11:23:51Z"),
"infoMessage" : "could not find member to sync from",
"configVersion" : 3,
"self" : true
}
],
"ok" : 1
"infoMessage" : "could not find member to sync from"
The primary and arbiter are both OK. I want to know the reason of this message and how to change the secondary's state from "recovering" to "secondary".