Im moving hundreds of topics from one broker to another. The process is
- Use
kafka-topics.sh
to generate existing partition list - Use
kafka-reassign-partitions.sh
to generate current list of partitions / brokers / etc - Edit this list so every instance of broker 7(to be replaced) is now brokers 7,4 (4 is new broker)
- Run
kafka-reassign-partitions.sh (broker list) --execute
to add new broker - Wait.. watch.. (using --verify).. until complete...
- Edit broker list to be brokers 4,7
- Do (4) again.. do (5) again..
- Run preferred leader election (in case 7 was leading anything)
- Edit broker list to remove all instances of broker 7
- Do (4) and (5) again
- Be happy
This has worked great for hundreds and hundreds of topics.. except for one sticky one. This holdout is refusing to sync up with the new broker (missing from ISR list) even though it's included in list of replicas
Output from kafka-topics.sh
(trying to replace broker 7 with broker 4):
Topic: shard_3 Partition: 7 Leader: 3 Replicas: 3,4,7 Isr: 7,3
I've run (4) above several times in hopes of getting this to complete but it doesn't seem to want to. I've waited overnight in case it's just really slow.
Suggestions on how to unstick this one?