When I'm using mongodump or mongorestore on a replica set these messages appear in the verbose output:
dbclient_rs replSetGetStatus failed{ ok: 0.0, errmsg: "unauthorized" }
The replica set is configured for authentication and the user account I'm connecting with has the clusterAdmin role set, which makes me wonder why the clients (mongodump/restore) aren't authorized to execute replSetGetStatus.
The connecting user has the userAdminAnyDatabase, readWriteAnyDatabase, dbAdminAnyDatabase and clusterAdmin roles on the admin database.
Here is an exemplified mongodump command:
mongodump --host rs0/host0.example.org,host1.example.org,host2.example.org \
--port 27017 \
--ssl \
--ipv6 \
--verbose \
--authenticationDatabase admin \
--username *user* \
--password *password* \
--out bakdir \
--db admin
mongod, mongodump and mongorestore are version 2.4.10, installed from the Debian Jessie (stable) repository.
I'd like to know why it fails, what the implications of these failings are and what I possibly can do about it (except switching off authentication ;-). Besides this quirk mongodump/restore seem to do their jobs just fine.
Here is the verbose log output from mongodump (timestamps truncated and hostnames replaced):
[Timestamp.381] starting new replica set monitor for replica set rs0 with seed of host0.example.org:27017,host1.example.org:27017,host2.example.org:27017
[Timestamp.381] creating new connection to:host0.example.org:27017
[Timestamp.383] BackgroundJob starting: ConnectBG
[Timestamp.412] connected connection!
[Timestamp.412] successfully connected to seed host0.example.org:27017 for replica set rs0
[Timestamp.413] ReplicaSetMonitor::_checkConnection: host0.example.org:27017 { setName: "rs0", ismaster: true, secondary: false, hosts: [ "host0.example.org:27017", "host2.example.org:27017", "host1.example.org:27017" ], primary: "host0.example.org:27017", me: "host0.example.org:27017", maxBsonObjectSize: 16777216, maxMessageSizeBytes: 48000000, localTime: new Date(1437578036410), ok: 1.0 }
[Timestamp.413] changing hosts to { 0: "host0.example.org:27017", 1: "host2.example.org:27017", 2: "host1.example.org:27017" } from rs0/
[Timestamp.413] trying to add new host host0.example.org:27017 to replica set rs0
[Timestamp.413] creating new connection to:host0.example.org:27017
[Timestamp.415] BackgroundJob starting: ConnectBG
[Timestamp.439] connected connection!
[Timestamp.439] successfully connected to new host host0.example.org:27017 in replica set rs0
[Timestamp.439] trying to add new host host1.example.org:27017 to replica set rs0
[Timestamp.439] creating new connection to:host1.example.org:27017
[Timestamp.440] BackgroundJob starting: ConnectBG
[Timestamp.448] connected connection!
[Timestamp.448] successfully connected to new host host1.example.org:27017 in replica set rs0
[Timestamp.448] trying to add new host host2.example.org:27017 to replica set rs0
[Timestamp.448] creating new connection to:host2.example.org:27017
[Timestamp.448] BackgroundJob starting: ConnectBG
[Timestamp.477] connected connection!
[Timestamp.477] successfully connected to new host host2.example.org:27017 in replica set rs0
[Timestamp.477] creating new connection to:host0.example.org:27017
[Timestamp.478] BackgroundJob starting: ConnectBG
[Timestamp.500] connected connection!
[Timestamp.501] dbclient_rs replSetGetStatus failed{ ok: 0.0, errmsg: "unauthorized" }
[Timestamp.501] _check : rs0/host0.example.org:27017,host1.example.org:27017,host2.example.org:27017
[Timestamp.501] ReplicaSetMonitor::_checkConnection: host0.example.org:27017 { setName: "rs0", ismaster: true, secondary: false, hosts: [ "host0.example.org:27017", "host2.example.org:27017", "host1.example.org:27017" ], primary: "host0.example.org:27017", me: "host0.example.org:27017", maxBsonObjectSize: 16777216, maxMessageSizeBytes: 48000000, localTime: new Date(1437578036498), ok: 1.0 }
[Timestamp.502] dbclient_rs replSetGetStatus failed{ ok: 0.0, errmsg: "unauthorized" }
[Timestamp.502] Primary for replica set rs0 changed to host0.example.org:27017
[Timestamp.503] ReplicaSetMonitor::_checkConnection: host0.example.org:27017 { setName: "rs0", ismaster: true, secondary: false, hosts: [ "host0.example.org:27017", "host2.example.org:27017", "host1.example.org:27017" ], primary: "host0.example.org:27017", me: "host0.example.org:27017", maxBsonObjectSize: 16777216, maxMessageSizeBytes: 48000000, localTime: new Date(1437578036500), ok: 1.0 }
[Timestamp.503] dbclient_rs replSetGetStatus failed{ ok: 0.0, errmsg: "unauthorized" }
[Timestamp.504] ReplicaSetMonitor::_checkConnection: host1.example.org:27017 { setName: "rs0", ismaster: false, secondary: true, hosts: [ "host1.example.org:27017", "host2.example.org:27017", "host0.example.org:27017" ], primary: "host0.example.org:27017", me: "host1.example.org:27017", maxBsonObjectSize: 16777216, maxMessageSizeBytes: 48000000, localTime: new Date(1437578036504), ok: 1.0 }
[Timestamp.505] creating new connection to:host1.example.org:27017
[Timestamp.506] BackgroundJob starting: ConnectBG
[Timestamp.514] connected connection!
[Timestamp.515] dbclient_rs replSetGetStatus failed{ ok: 0.0, errmsg: "unauthorized" }
[Timestamp.516] ReplicaSetMonitor::_checkConnection: host2.example.org:27017 { setName: "rs0", ismaster: false, secondary: true, hosts: [ "host2.example.org:27017", "host1.example.org:27017", "host0.example.org:27017" ], primary: "host0.example.org:27017", me: "host2.example.org:27017", maxBsonObjectSize: 16777216, maxMessageSizeBytes: 48000000, localTime: new Date(1437578036516), ok: 1.0 }
[Timestamp.516] creating new connection to:host2.example.org:27017
[Timestamp.516] BackgroundJob starting: ConnectBG
[Timestamp.544] connected connection!
[Timestamp.544] dbclient_rs replSetGetStatus failed{ ok: 0.0, errmsg: "unauthorized" }
[Timestamp.544] replica set monitor for replica set rs0 started, address is rs0/host0.example.org:27017,host1.example.org:27017,host2.example.org:27017
connected to: rs0/host0.example.org,host1.example.org,host2.example.org:27017
[Timestamp.545] creating new connection to:host0.example.org:27017
[Timestamp.545] BackgroundJob starting: ReplicaSetMonitorWatcher
[Timestamp.545] [ReplicaSetMonitorWatcher] starting
[Timestamp.546] BackgroundJob starting: ConnectBG
[Timestamp.571] connected connection!
[Timestamp.573] creating new connection to:host1.example.org:27017
[Timestamp.574] BackgroundJob starting: ConnectBG
[Timestamp.582] connected connection!
[Timestamp.585] DATABASE: admin to ./admin
[Timestamp.589] admin.system.indexes to ./admin/system.indexes.bson
[Timestamp.591] 2 objects
[Timestamp.591] skipping collection: admin.system.users.$user_1_userSource_1
[Timestamp.591] skipping collection: admin.system.users.$_id_
[Timestamp.591] admin.system.users to ./admin/system.users.bson
[Timestamp.593] 2 objects
[Timestamp.593] Metadata for admin.system.users to ./admin/system.users.metadata.json
[Timestamp.593] admin.system.profile to ./admin/system.profile.bson
[Timestamp.646] 2612 objects
[Timestamp.646] Metadata for admin.system.profile to ./admin/system.profile.metadata.json
And here are the corresponding mongodb.log files for the replica set servers. Host addresses and user credentials replaced. Timestamps truncated except milliseconds. Mongodump was executed on host2. Backup was taken from host1. The SocketExceptions in the server logs might be noteworthy.
host0.example.org (primary):
[Timestamp.381] [initandlisten] connection accepted from <backup.client.ipv4.addr>:40668 #46757 (8 connections now open)
[Timestamp.410] [conn46757] runQuery called admin.$cmd { ismaster: 1 }
[Timestamp.410] [conn46757] run command admin.$cmd { ismaster: 1 }
[Timestamp.410] [conn46757] command admin.$cmd command: { ismaster: 1 } ntoreturn:1 keyUpdates:0 reslen:341 0ms
[Timestamp.413] [initandlisten] connection accepted from <backup.client.ipv4.addr>:40669 #46758 (9 connections now open)
[Timestamp.416] [journal] journal WRITETODATAFILES 0.144ms
[Timestamp.476] [initandlisten] connection accepted from <backup.client.ipv4.addr>:40672 #46759 (10 connections now open)
[Timestamp.497] [conn46759] runQuery called admin.$cmd { replSetGetStatus: 1 }
[Timestamp.497] [conn46759] run command admin.$cmd { replSetGetStatus: 1 }
[Timestamp.497] [conn46759] command denied: { replSetGetStatus: 1 }
[Timestamp.497] [conn46759] command admin.$cmd command: { replSetGetStatus: 1 } ntoreturn:1 keyUpdates:0 reslen:62 0ms
[Timestamp.498] [conn46757] SocketException: remote: <backup.client.ipv4.addr>:40668 error: 9001 socket exception [CLOSED] server [<backup.client.ipv4.addr>:40668]
[Timestamp.498] [conn46757] end connection <backup.client.ipv4.addr>:40668 (9 connections now open)
[Timestamp.498] [conn46758] runQuery called admin.$cmd { ismaster: 1 }
[Timestamp.498] [conn46758] run command admin.$cmd { ismaster: 1 }
[Timestamp.498] [conn46758] command admin.$cmd command: { ismaster: 1 } ntoreturn:1 keyUpdates:0 reslen:341 0ms
[Timestamp.499] [conn46759] runQuery called admin.$cmd { replSetGetStatus: 1 }
[Timestamp.499] [conn46759] run command admin.$cmd { replSetGetStatus: 1 }
[Timestamp.499] [conn46759] command denied: { replSetGetStatus: 1 }
[Timestamp.499] [conn46759] command admin.$cmd command: { replSetGetStatus: 1 } ntoreturn:1 keyUpdates:0 reslen:62 0ms
[Timestamp.500] [conn46758] runQuery called admin.$cmd { ismaster: 1 }
[Timestamp.500] [conn46758] run command admin.$cmd { ismaster: 1 }
[Timestamp.500] [conn46758] command admin.$cmd command: { ismaster: 1 } ntoreturn:1 keyUpdates:0 reslen:341 0ms
[Timestamp.500] [conn46759] runQuery called admin.$cmd { replSetGetStatus: 1 }
[Timestamp.500] [conn46759] run command admin.$cmd { replSetGetStatus: 1 }
[Timestamp.500] [conn46759] command denied: { replSetGetStatus: 1 }
[Timestamp.501] [conn46759] command admin.$cmd command: { replSetGetStatus: 1 } ntoreturn:1 keyUpdates:0 reslen:62 0ms
[Timestamp.544] [initandlisten] connection accepted from <backup.client.ipv6.addr>:56132 #46760 (10 connections now open)
[Timestamp.569] [conn46760] runQuery called admin.$cmd { getnonce: 1 }
[Timestamp.569] [conn46760] run command admin.$cmd { getnonce: 1 }
[Timestamp.569] [conn46760] command admin.$cmd command: { getnonce: 1 } ntoreturn:1 keyUpdates:0 reslen:65 0ms
[Timestamp.569] [conn46760] runQuery called admin.$cmd { authenticate: 1, nonce: "**nonce**", user: "**user**", key: "**key**" }
[Timestamp.569] [conn46760] run command admin.$cmd { authenticate: 1, nonce: "**nonce**", user: "**user**", key: "**key**" }
[Timestamp.569] [conn46760] authenticate db: admin { authenticate: 1, nonce: "**nonce**", user: "**user**", key: "**key**" }
[Timestamp.570] [conn46760] user:**user**@admin command admin.$cmd command: { authenticate: 1, nonce: "**nonce**", user: "**user**", key: "**key**" } ntoreturn:1 keyUpdates:0 locks(micros) r:235 reslen:79 0ms
[Timestamp.623] [journal] journal WRITETODATAFILES 0.073ms
[Timestamp.648] [conn46758] SocketException: remote: <backup.client.ipv4.addr>:40669 error: 9001 socket exception [CLOSED] server [<backup.client.ipv4.addr>:40669]
[Timestamp.648] [conn46758] end connection <backup.client.ipv4.addr>:40669 (9 connections now open)
[Timestamp.649] [conn46759] SocketException: remote: <backup.client.ipv4.addr>:40672 error: 9001 socket exception [CLOSED] server [<backup.client.ipv4.addr>:40672]
[Timestamp.649] [conn46759] end connection <backup.client.ipv4.addr>:40672 (8 connections now open)
[Timestamp.649] [conn46760] user:**user**@admin SocketException: remote: <backup.client.ipv6.addr>:56132 error: 9001 socket exception [CLOSED] server [<backup.client.ipv6.addr>:56132]
[Timestamp.649] [conn46760] user:**user**@admin end connection <backup.client.ipv6.addr>:56132 (7 connections now open)
[Timestamp.677] [conn46756] user:**user**@admin SocketException: remote: <backup.client.ipv6.addr>:56124 error: 9001 socket exception [CLOSED] server [<backup.client.ipv6.addr>:56124]
[Timestamp.677] [conn46756] user:**user**@admin end connection <backup.client.ipv6.addr>:56124 (6 connections now open)
host1.example.org (secondary & host from which the backup was taken):
[Timestamp.442] [initandlisten] connection accepted from <backup.client.ipv4.addr>:33206 #46856 (3 connections now open)
[Timestamp.504] [conn46856] runQuery called admin.$cmd { ismaster: 1 }
[Timestamp.504] [conn46856] run command admin.$cmd { ismaster: 1 }
[Timestamp.504] [conn46856] command admin.$cmd command: { ismaster: 1 } ntoreturn:1 keyUpdates:0 reslen:341 0ms
[Timestamp.507] [initandlisten] connection accepted from <backup.client.ipv4.addr>:33209 #46857 (4 connections now open)
[Timestamp.515] [conn46857] runQuery called admin.$cmd { replSetGetStatus: 1 }
[Timestamp.515] [conn46857] run command admin.$cmd { replSetGetStatus: 1 }
[Timestamp.515] [conn46857] command denied: { replSetGetStatus: 1 }
[Timestamp.515] [conn46857] command admin.$cmd command: { replSetGetStatus: 1 } ntoreturn:1 keyUpdates:0 reslen:62 0ms
[Timestamp.575] [initandlisten] connection accepted from <backup.client.ipv6.addr>:60931 #46858 (5 connections now open)
[Timestamp.582] [conn46858] runQuery called admin.$cmd { getnonce: 1 }
[Timestamp.582] [conn46858] run command admin.$cmd { getnonce: 1 }
[Timestamp.582] [conn46858] command admin.$cmd command: { getnonce: 1 } ntoreturn:1 keyUpdates:0 reslen:65 0ms
[Timestamp.583] [conn46858] runQuery called admin.$cmd { authenticate: 1, nonce: "**nonce**", user: "**user**", key: "**key**" }
[Timestamp.583] [conn46858] run command admin.$cmd { authenticate: 1, nonce: "**nonce**", user: "**user**", key: "**key**" }
[Timestamp.583] [conn46858] authenticate db: admin { authenticate: 1, nonce: "**nonce**", user: "**user**", key: "**key**" }
[Timestamp.584] [conn46858] user:**user**@admin command admin.$cmd command: { authenticate: 1, nonce: "**nonce**", user: "**user**", key: "**key**" } ntoreturn:1 keyUpdates:0 locks(micros) r:393 reslen:79 0ms
[Timestamp.585] [conn46858] user:**user**@admin runQuery called admin.$cmd { isdbgrid: 1 }
[Timestamp.585] [conn46858] user:**user**@admin run command admin.$cmd { isdbgrid: 1 }
[Timestamp.585] [conn46858] user:**user**@admin command admin.$cmd command: { isdbgrid: 1 } ntoreturn:1 keyUpdates:0 reslen:99 0ms
[Timestamp.586] [conn46858] user:**user**@admin runQuery called admin.system.indexes {}
[Timestamp.586] [conn46858] user:**user**@admin query admin.system.indexes ntoreturn:0 ntoskip:0 nscanned:2 keyUpdates:0 locks(micros) r:49 nreturned:2 reslen:207 0ms
[Timestamp.587] [conn46858] user:**user**@admin runQuery called admin.system.namespaces {}
[Timestamp.587] [conn46858] user:**user**@admin query admin.system.namespaces ntoreturn:0 ntoskip:0 nscanned:5 keyUpdates:0 locks(micros) r:39 nreturned:5 reslen:254 0ms
[Timestamp.589] [conn46858] user:**user**@admin runQuery called admin.$cmd { count: "system.indexes", query: {} }
[Timestamp.589] [conn46858] user:**user**@admin run command admin.$cmd { count: "system.indexes", query: {} }
[Timestamp.589] [conn46858] user:**user**@admin command admin.$cmd command: { count: "system.indexes", query: {} } ntoreturn:1 keyUpdates:0 locks(micros) r:17 reslen:48 0ms
[Timestamp.590] [conn46858] user:**user**@admin runQuery called admin.$cmd { availablequeryoptions: 1 }
[Timestamp.590] [conn46858] user:**user**@admin run command admin.$cmd { availablequeryoptions: 1 }
[Timestamp.590] [conn46858] user:**user**@admin command admin.$cmd command: { availablequeryoptions: 1 } ntoreturn:1 keyUpdates:0 reslen:50 0ms
[Timestamp.591] [conn46858] user:**user**@admin runQuery called admin.system.indexes { query: {}, $snapshot: true }
[Timestamp.591] [conn46858] user:**user**@admin query admin.system.indexes query: { query: {}, $snapshot: true } ntoreturn:0 ntoskip:0 nscanned:2 keyUpdates:0 locks(micros) r:25 nreturned:2 reslen:207 0ms
[Timestamp.592] [conn46858] user:**user**@admin runQuery called admin.$cmd { count: "system.users", query: {} }
[Timestamp.592] [conn46858] user:**user**@admin run command admin.$cmd { count: "system.users", query: {} }
[Timestamp.592] [conn46858] user:**user**@admin command admin.$cmd command: { count: "system.users", query: {} } ntoreturn:1 keyUpdates:0 locks(micros) r:13 reslen:48 0ms
[Timestamp.593] [conn46858] user:**user**@admin runQuery called admin.system.users { query: {}, $snapshot: true }
[Timestamp.593] [conn46858] user:**user**@admin query admin.system.users query: { query: {}, $snapshot: true } ntoreturn:0 ntoskip:0 nscanned:2 keyUpdates:0 locks(micros) r:79 nreturned:2 reslen:350 0ms
[Timestamp.594] [conn46858] user:**user**@admin runQuery called admin.$cmd { count: "system.profile", query: {} }
[Timestamp.594] [conn46858] user:**user**@admin run command admin.$cmd { count: "system.profile", query: {} }
[Timestamp.594] [conn46858] user:**user**@admin command admin.$cmd command: { count: "system.profile", query: {} } ntoreturn:1 keyUpdates:0 locks(micros) r:14 reslen:48 0ms
[Timestamp.595] [conn46858] user:**user**@admin runQuery called admin.system.profile { query: {}, $snapshot: true }
[Timestamp.595] [conn46858] user:**user**@admin query admin.system.profile query: { query: {}, $snapshot: true } cursorid:**cursorid** ntoreturn:0 ntoskip:0 exhaust:1 nscanned:102 keyUpdates:0 locks(micros) r:163 nreturned:101 reslen:38501 0ms
[Timestamp.597] [conn46858] user:**user**@admin getmore admin.system.profile query: { query: {}, $snapshot: true } cursorid:**cursorid** ntoreturn:0 keyUpdates:0 locks(micros) r:1739 nreturned:2511 reslen:960456 1ms
[Timestamp.651] [conn46857] SocketException: remote: <backup.client.ipv4.addr>:33209 error: 9001 socket exception [CLOSED] server [<backup.client.ipv4.addr>:33209]
[Timestamp.651] [conn46857] end connection <backup.client.ipv4.addr>:33209 (4 connections now open)
[Timestamp.651] [conn46856] SocketException: remote: <backup.client.ipv4.addr>:33206 error: 9001 socket exception [CLOSED] server [<backup.client.ipv4.addr>:33206]
[Timestamp.651] [conn46856] end connection <backup.client.ipv4.addr>:33206 (3 connections now open)
[Timestamp.652] [conn46858] user:**user**@admin SocketException: remote: <backup.client.ipv6.addr>:60931 error: 9001 socket exception [CLOSED] server [<backup.client.ipv6.addr>:60931]
[Timestamp.652] [conn46858] user:**user**@admin end connection <backup.client.ipv6.addr>:60931 (2 connections now open)
host2.example.org (secondary & backup-client):
[Timestamp.449] [initandlisten] connection accepted from <backup.client.ipv4.addr>:58052 #46602 (4 connections now open)
[Timestamp.474] [conn46601] user:__system@local runQuery called admin.$cmd { replSetHeartbeat: "rs0", v: 4, pv: 1, checkEmpty: false, from: "host1.example.org:27017" }
[Timestamp.475] [conn46601] user:__system@local run command admin.$cmd { replSetHeartbeat: "rs0", v: 4, pv: 1, checkEmpty: false, from: "host1.example.org:27017" }
[Timestamp.475] [conn46601] user:__system@local command: { replSetHeartbeat: "rs0", v: 4, pv: 1, checkEmpty: false, from: "host1.example.org:27017" }
[Timestamp.475] [conn46601] user:__system@local command admin.$cmd command: { replSetHeartbeat: "rs0", v: 4, pv: 1, checkEmpty: false, from: "host1.example.org:27017" } ntoreturn:1 keyUpdates:0 reslen:164 0ms
[Timestamp.516] [conn46602] runQuery called admin.$cmd { ismaster: 1 }
[Timestamp.516] [conn46602] run command admin.$cmd { ismaster: 1 }
[Timestamp.516] [conn46602] command admin.$cmd command: { ismaster: 1 } ntoreturn:1 keyUpdates:0 reslen:334 0ms
[Timestamp.516] [initandlisten] connection accepted from <backup.client.ipv4.addr>:58055 #46603 (5 connections now open)
[Timestamp.544] [conn46603] runQuery called admin.$cmd { replSetGetStatus: 1 }
[Timestamp.544] [conn46603] run command admin.$cmd { replSetGetStatus: 1 }
[Timestamp.544] [conn46603] command denied: { replSetGetStatus: 1 }
[Timestamp.544] [conn46603] command admin.$cmd command: { replSetGetStatus: 1 } ntoreturn:1 keyUpdates:0 reslen:62 0ms
[Timestamp.650] [conn46602] SocketException: remote: <backup.client.ipv4.addr>:58052 error: 9001 socket exception [CLOSED] server [<backup.client.ipv4.addr>:58052]
[Timestamp.651] [conn46602] end connection <backup.client.ipv4.addr>:58052 (4 connections now open)
[Timestamp.651] [conn46603] SocketException: remote: <backup.client.ipv4.addr>:58055 error: 9001 socket exception [CLOSED] server [<backup.client.ipv4.addr>:58055]
[Timestamp.651] [conn46603] end connection <backup.client.ipv4.addr>:58055 (3 connections now open)