We are currently running two Kafka clusters. One is inside our corporate network and another is in AWS. We need to stream data from the internal Kafka to the cluster in AWS. Mirror Maker is currently our first choice based on simplicity. I've been investigating on how to do this in a secure fashion. I would prefer not to place Kafka on a public subnet as it seems that is not a secure option but, some limitations of Kafka have made this somewhat difficult. Namely that Kafka producers/consumers need to be able to see all instances of the target cluster on the network. This means that all Kafka instances in AWS need a public IP address. The alternative is to place Mirror Maker in AWS, but then how to we expose our internal servers to the web in a secure manner? This seems like it might be a common use case, but I cannot find anything on the web that relates. Can someone provide recommendations or correct me if any of my assumptions are incorrect?
↧