I am testing Couchbase 3.1's XDCR feature and keep got the error message as described below:
- environment
6 vm for cluster1
6 vm for cluster2
uni-directional XDCR from cluster1 to cluster2.
There are about 900 MB testing data in both two clusters.
- From the Source(Cluster1) 's XDCR monitoring UI(The error messages and logs are only shown up on the source cluster), it tells the XDCR is ongoing but with errors, the "Status" column shows " Replicating Last 10 errors".
After click the "Last 10 errors", a windows shown up and many similar errors on it .
"Error replicating vbucket XXX. Please see logs for details."
- Then i checked ns_server.xdcr_errors.log and found a lots of errors like below:
[xdcr:error,2015-09-10T14:02:07.638,ns_1@1.2.3.4:<0.530.0>:xdc_vbucket_rep:terminate:497]Replication (XMem mode)
8aaa3c6dc842ba99edeb624914e1a092/src-bkt01/dest-bkt01(src-bkt01/989->http://*****@4.3.2.1:8092/dest-bkt01%2B9caf5f56faf1f993b81e7654a3b421b3%2f989) failed.Please see ns_server debug log for complete state dump
- Then i opened ns_server.debug.log to check and found below errors:
[error_logger:error,2015-09-10T14:03:56.705,ns_1@1.2.3.4:error_logger<0.6.0>:ale_error_logger_handler:do_log:203]** Generic server <0.10121.312> terminating ** Last message in was {'EXIT',<0.12931.330>, {failed_write, {"in batch of 1 docs: flushed: 0, rejected (eexists): 0; remote memcached errors: enoent: 0, not-my-vb: 1, invalid: 0, tmp fail: 0, enomem: 0, others: 0", "Error with their keys:1 keys with not_my_vbucket errors([<<\"userEventCntrect.xwf-id.com2015092300000009\">>]); "}}}
I have checked the total amount of items on both sides and the two clusters are keeping up with the same number of items.
Also i have checked couchbase's issues site with the error msg "not_my_vbucket" try to figure out what does it really means to my system's problem.
Appreciate if any opinions and suggestion.
Joe