I'm having a strange issue with some Cassandra clusters on a set of Solaris servers. Each of the clusters has 3 servers with its replication factor set to 3.
[admin@unknown] describe resolve;
WARNING: CQL3 tables are intentionally omitted from 'describe' output.
See https://issues.apache.org/jira/browse/CASSANDRA-4377 for details.
Keyspace: resolve:
Replication Strategy: org.apache.cassandra.locator.SimpleStrategy
Durable Writes: true
Options: [replication_factor:3]
Column Families:
But when we run the "nodetool ring" command it is reporting that each server owns only 33.33% of the data.
Datacenter: datacenter1
==========
Address Rack Status State Load Owns Token
113427455640312821154458202477256070484
10.46.36.187 rack1 Up Normal 44.78 GB 33.33% 0
10.46.36.189 rack1 Up Normal 39.59 GB 33.33% 56713727820156410577229101238628035242
10.46.36.190 rack1 Up Normal 34.78 GB 33.33% 113427455640312821154458202477256070484
In all other clusters with the same settings they report 100% ownership. From this it appears that the replication factor being used is still 1. The other odd thing is that the nodetool ring is not printing out the "Replicas" line when our other clusters do. All our clusters are currently version 1.2.5.
I've tried running the "nodetool repair" command on all the nodes and re-ran the "update keyspace" command to set the replication_factor but the ownership percentage remains unchanged. Is there anything else I can look at or check to see why this is happening?
Edit:
This is what I normally see in my other clusters:
Datacenter: datacenter1
==========
Replicas: 3
Address Rack Status State Load Owns Token
113427455640312821154458202477256070484
10.50.2.65 rack1 Up Normal 126.65 KB 100.00% 0
10.50.2.66 rack1 Up Normal 122.15 KB 100.00% 56713727820156410577229101238628035242
10.50.2.67 rack1 Up Normal 122.29 KB 100.00% 113427455640312821154458202477256070484