Automated Deployment with Apache Kafka

It’s likely not a surprise that Cerner would use Apache Kafka as we have used a number of related technologies like Apache Hadoop along with its Map/Reduce, HDFS and even Apache HBase. Our team first started using Apache Kafka in 2014 when Kafka 0.8 first came out. Since then we’ve expanded to using Kafka for a number of different use cases (1, 2) and it has become a core piece of Cerner’s infrastructure.

Just like the applications we create, we also needed to automate the deployment of Kafka to handle the ever growing amount of operations work and to ensure a high level of consistency with our environments. Chef was an obvious choice for us since its been our deployment tool of choice for the last few years. We quickly put together a cookbook to help us automate the deployment of the many Kafka clusters here at Cerner. We have continued to support and use this cookbook after open-sourcing it upgrading it to handle newer versions of Kafka (0.9, 0.10), handle Kerberos authentication, and many other improvements.

If you use Apache Kafka, feel free to try it out and let us know how it works for you.