February 6, 2018

258 words 2 mins read

Cerner Open Sources its Kafka Utilities

At Cerner, we often make use of many open source projects in our infrastructure. I work on a team responsible for Cerner’s Ingestion Platform, a critical piece of infrastructure that takes in TBs of data and over a billion messages per day. The platform’s responsibility is then to make this data available for downstream teams to consume. When designing the Ingestion Platform, we felt Apache Kafka was perfect for ingesting and consuming these massive streams of data. We originally built the Ingestion Platform in 2014 and it has grown with Kafka ever since.

Along the way, we’ve become experienced with Apache Kafka and how it works. We know how to monitor it, perform common administrative operations, and more. We’ve even made contributions to Kafka for various improvements [1][2][3] and Apache Crunch, our processing framework, to add Kafka support. We have also given talks about our usage of Apache Kafka at Kafka Summit and we’ve written a blog post about it as well.

We are still interested in supporting the Kafka community, and today we hit another milestone as we open source some of our utilities for Kafka. This includes:

  • Lightweight wrappers over producer and consumer providing additional functionality
  • A client to simplify admin operations
  • A Kafka connect sink for delivering data to another Kafka cluster
  • Kafka test infrastructure for integration testing

Try it and let me know what you think! You can log issues for improvements, or email me your thoughts: bryan.baugher@cerner.com. We hope to continue to improve the library and always welcome contributions and ideas from others.

engineering kafka ingestion

Automated Deployment with Apache Kafka

February 16, 2017

It’s likely not a surprise that Cerner would use Apache Kafka as we have used a number of related technologies like Apache Hadoop along with its Map/Reduce, HDFS and even Apache HBase. Our team first started using Apache Kafka in 2014 when Kafka 0.8 first came out. Since then we’ve expanded to using Kafka for a number of different use cases (1, 2) and it has become a core piece of Cerner’s infrastructure.

Cerner and the SDLC

January 23, 2018

Cerner has a very compelling story that many want to learn from. Cerner’s Agile Champions are regular presenters at local, national, and global conferences. We routinely host calls and site visits for interested companies around the world who want to figure out how we were able to adopt Agile so quickly and sustain it so successfully. Despite our success, there has been a general feeling at the engineering leadership and Agile Champion tiers that we aren’t getting the benefits from Agile that we once were.

2^5 Coding Competition 2017: 32 lines or less

December 19, 2017

The Competition Part of Cerner’s engineering culture is to celebrate technology. This past fall, we had our annual celebration for Programmers’ Day. We celebrated the day with great food (taco bar and ice cream) and an evening of trivia. Topping off Programmers' Day with a taco bar and trivia night with @geekswhodrink pic.twitter.com/pSjqBVBHxK — Cerner Engineering (@CernerEng) September 13, 2017 Happy Programmers' Day! Celebrating with an ice cream social. pic.