We are huge believers of ChatOps at VividCortex. If you are unfamiliar with the term “ChatOps”, it was coined by GitHub, and the core idea is simple: make common commands executable from within your company’s chatroom via a chatbot. (GitHub opensourced their chatbot Hubot.) This allows everyone in that room to see you execute those commands, making the commands immediately accessible. New hires learn commands by seeing current employees execute them regularly. These simple concepts turn out to be very powerful. GitHub’s implementation of ChatOps is amazing and has inspired me to spearhead ChatOpts within our office.
ChatOps has really revolutionized how we operate. It surprised me that outside of GitHub, it appears ChatOps has not gained much traction. It would be a shame if ChatOps is simply commands to get random funny images at other companies. At VividCortex, ChatOps is a tool to help make everyone more productive plus funny random images. As the unofficial company Hubot Trainer, I made it my mission to teach our Hubot tasks that would help everyone. Here are some of the problems I found in the company and how I solved them with ChatOps.
Teach Hubot How To Deploy Code
The first problem I sought to solve was improving how we deploy code. The problem at the time was developers were not pushing code to stage or production very often, mainly because they needed Capistrano (a commandline based deployment tool) installed on their machine to do so. Additionally, their ssh key needed to be copied to the remote server where the code would be deployed. This seems simple enough but is not always easy to get working correctly. For example, if the process of setting those tools up is painful for the new hire, a bad first impression and a general fear of deploying is created. Keep in mind that the new hire is trying to get accustomed to the code base and mucking around with installing and setting up these tools is taking away from their time. Combine that with the awful setup process, and the new hire opts to delegate the task of running this command to someone who has it installed. That helpful developer becomes the gatekeeper and one of a few who can decide when code gets deployed. This regular interruption not only impacts their productivity since being a gatekeeper is not their full time position, but also causes deploying to happen less often than it could. In our case, it probably was only a few times a week.
To make the process of deploying code better, I began by moving the execution of the Capistrano commands to a Jenkins (a continuous integration server) job. Next, I created Hubot commands to start those jobs using the REST API that Jenkins exposes. With a command to deploy, I showed our other developers how to execute it by running it in the chatroom
hubot stage webapp or to deploy
hubot release webapp. Now, no one is blocked by the gatekeeper from pushing code into staging or production, and every developer can push code when they are ready. The unfamiliar Capistrano commands were simplified into much easier to understand Hubot commands. New hires can spend more time getting accustomed with the code base rather than struggling with the setup of Capistrano and SSH keys. They quickly gain confidence in deploying by seeing others in the room deploy. In the case something does go wrong with a deployment, Hubot posts back a link to the console output from Jenkins. When I see that or someone pings me in the chatroom telling me their build has failed, I can click on the link to see the console output. This saves me a ton of time trying to reproduce the problem locally, enabling me to quickly jump in and help resolve the problem.
Teach Hubot Bookmarks
The next problem is discovering and remembering the urls to various applications and websites that we have at VividCortex. While a company-wide wiki or google doc that has all those links defined is helpful, there is still the problem of discovery. You have to visually grep through the document to find the appropriate link. Inevitably, you will ping another employee, causing an interruption and subsqequent loss of productivity. With a few employees this is probably not a big deal, but as the company grows, the problem is magnified.
The solution to this is very simple. I created a Hubot script to store static links, mapping those links to simple, easy to remember commands. For example
hubot waffle returns the url to our Waffle.io Kanban board. Or
hubot kibana to get quick access to our Kibana instance. If an employee does not remember the link, they can use the built-in Hubot-help command
hubot help to get a list of all the links we have.
Teach Hubot How To Fetch Logs
The architecture at VividCortex is best described as a Service Oriented Architecture or SOA. Rather than have a single monolithic application, we have many smaller applications, typically spread over a number of servers. This type of architecture tends to scale better than its monolithic cousin but makes it more difficult to trace and debug errors. The developer needs to ssh into a number of servers and tail logs in the attempt to debug the issue at hand. This is where log aggregation comes into play. We use Fluentd to aggregate and parse our logs. Then we send them to Elasticsearch to make them searchable, and on top of Elasticsearch, we use Kibana to visually analyze them.
Suppose a problem with one of our production applications begins. Typically, one of us will notice the problem or a customer will become aware, and conversation about that problem will erupt in the chatroom. The developer should not have to leave the chatroom to fetch the logs; they should be able to pull that data into the conversation with a Hubot command. Given that all our log data is in Elasticsearch I was able to make use of the great REST API Elasticsearch provides. By wrapping those API calls in a Hubot command, a developer from within the chatroom can execute
hubot log me foo:bar. For example, if you want the last 25 entries from a specific host, you can run
hubot log me host:apistage. By pulling that data into the chatroom, the hope is they can diagnose the issue without having to leave, saving the developer the time it takes to tunnel around servers looking for logs.
Teach Hubot How To Be Funny
All work and no play makes Hubot a dull robot. While some might say that commands to get random funny cat pictures and stupid videos is not very important and a waste, I believe it serves a very useful purpose - employee moral! It is all too easy to burn out as a developer. While it is not physically exerting work, it is mentally exhausting. One of the best cures for stress and mental fatigue is a good laugh. Plus, we want our developers to hang out in the chatroom. If it is a fun place, they are more likely to do so. Here are a couple of the funny Hubot commands we created in house.
At VividCortex, we sure do love our cats; our development team collectively owns as many cats as we have developers. This is probably why we have a command called
hubot cats me (source) which gets a random cat picture provided by The Cat API. This command also has a bomb feature
hubot cats bomb which will give you 5 random cat pictures.
We understand it is not easy being a DBA, so we sympathize with the problems they face. That is why we are huge fans of the website http://dbareactions.com/. Turns out this is a Tumblr blog from which you can request the 50 most recent posts as JSON. Using that data, I created a Hubot command called
hubot dba me (source) which will pick at random one of the 50 most recent posts and return the image and caption to our chatroom.
Finally there is
hubot hold my beer or
hubot hmb (source) for short. This command talks to the Reddit API and pulls down one of this week’s top images or videos from the Hold My Beer feed. This feed consists of spectacular acts of human failure, and the command works best when you feel like a failure. At least you do not feel like a complete failure after seeing one of these videos or images.
I hope you enjoyed my overview of ChatOps at VividCortex. So far it has proven to be an excellent choice for the problems I discussed above, and we believe that we have only scratched the surface of what ChatOps can be. Further, I hope this post inspires you to adopt ChatOps within your organization or share your story if you already have.