November 17, 2015

371 words 2 mins read

Identifying Network ACL Issues with Chef Locally

At Cerner, we use Chef to automate our deployments across a very large distributed system comprised of many thousands of nodes. While we employ ServerSpec and Test Kitchen extensively to test our Chef recipes, this still doesn’t identify all possible failure scenarios. In a large distributed system such as ours, we occasionally run into connectivity failures between machines due to network ACL issues. This could be due to a variety of reasons. The simplest being that you didn’t know what all you were connecting to. The second being that you didn’t ask for the ACLs to be opened early enough, or at all. And finally (and most frustrating), you asked and were granted an ACL change but there still exists an issue (like an unfollowed redirect). When we took a look at these issues, we saw an opportunity for improvement in our Chef testing.

This is why we created ops_tcpdump_handler, an open source Chef cookbook that is helping us identify network connectivity issues during Chef test runs.

As explained in the project README, when you add this cookbook into a Test Kitchen or Vagrant run list, you’ll see an output similar to below that shows connection attempts between nodes. This will not only help you understand what nodes you’re connecting to in a Chef run, but also diagnose any connectivity issues that arise.

==> node01: attempted to connect to: proxy06.fedoraproject.org using http
==> node01: attempted to connect to: mirror.sfo12.us.leaseweb.net using http
==> node01: attempted to connect to: li63-48.members.linode.com using https
==> node01: attempted to connect to: 23.235.40.133 using https
==> node01: attempted to connect to: github.com using https
==> node01: attempted to connect to: codeload.github.com using https
==> node01: attempted to connect to: s3-1.amazonaws.com using https

We’re currently using this cookbook in our Chef test runs to identify and help troubleshoot connectivity issues between nodes. Next, we’re going to incorporate this into our continuous delivery pipeline to verify connectivity first before proceeding with the deployment. We decided to open source this work to get feedback from the community and to hear your experience with these types of issues. We’d love to hear your experience with incorporating this cookbook into your pipeline or other approaches you’ve taken with these types of issues.

engineering
JavaScript Logging: We can do better!

JavaScript Logging: We can do better!

July 20, 2015

Currently in the world of JavaScript these options are what we most commonly use to generate logs: console.log console.info console.warn console.error These are actually pretty good in most modern browsers. Even if you go back to Internet Explorer 8 console.log and friends work as long as you have the developer tools open. Given that we have these logging utilities what is the problem with using them? When in local development these are just fine for helping debug and speed up development.
Managing 30,000 Logging Events per Day with Splunk

Managing 30,000 Logging Events per Day with Splunk

January 29, 2015

Our team works on a patient facing web application with a thousand live clients with 2,315,000+ users. On an average, the high traffic results into more than 40,000 visits and 300,000 page views daily generating about 30,000 logging events. A considerable portion of these events are of information and warning level in order to aid proactive monitoring or identify potential issues due to clients’ misconfiguration. Before Splunk To handle this large volume of logging, our team created a rotational support role to manually monitor the logs at regular intervals daily.
Cerner and the Apache Software Foundation

Cerner and the Apache Software Foundation

October 28, 2014

At the beginning of this year, we announced that Cerner became a bronze-level sponsor of the non-profit Apache Software Foundation (ASF). Many of the open source projects we use and contribute to are under the ASF umbrella, so supporting the mission and work of the ASF is important to us. We’re happy to announce that Cerner has now increased our sponsorship of the ASF to become a silver-level sponsor. Open source continues to play an integral role in both our architecture and engineering culture.