The Brainiac Corner is a format where we talk with some of the smartest minds in the system, database, devops, and IT world. If you have opinions on pirates, or anything else related, please don’t hesitate to contact us.
Today, we interview Jeremy Tinley, the current Database Administrator at Etsy. He is a huge fan of puns and jokes. His favorite today is “A DBA walks into a bar and sees two tables. He says ‘Can I join you?’” He gets randomly preselected for TSA precheck lines more than anyone else he knows. If you are interested in his thoughts, connect through Twitter.
How did you get from stork to brainiac (i.e. what do you do today and how did you get there)?
My first exposure to computers was a PC running DOS and Windows 3.1. I started writing batch scripts to make DOS more bearable. From there, I got into ISP technical support, learning about the basic Internet protocols and eventually taught myself Perl and the LAMP stack. I found a love of MySQL both because of its function but also its challenges. Back in the old days, no one had elegant solutions for HA like Galera, so we rolled our own using DRBD and Heartbeat. Today, I’m involved in running MySQL at scale for Etsy. There are so many challenges MySQL still faces, from monitoring to automation. I love a good challenge.
What is in your group’s technology stack?
Externally, the pretty typical LAMP(HP) stack with a memcache caching layer and a Gearman worker system. Nagios is the monitoring system of choice. We also use Ganglia, Graphite, Logstash, Splunk and a number of other tools to make our life easier. We have also released several open source projects: most people know about statsd, but we have some other great projects like nagios-herald, which adds context to your Nagios emails and opsweekly, a tool for classifying alerts and tracking on-call life.
Who would win in a fight between ninjas and pirates? Why?
Ninjas. Pirates are awesome on their own, but in a fight, ninjas would have the element of surprise. Also: scurvy.
**Which is a more accurate state of the world, #monitoringsucks or #monitoringlove? **
I’m in #monitoringlove. Etsy has an immense monitoring and graphing culture, detailed by Daniel Schauenberg’s excellent Monitorama talk. We have dashboards of dashboards of dashboards. It’s almost overwhelming the amount of data we collect, but the incredible part is that you can whip up a new graph or dashboard to answer any questions that come up. We recently had a case where we saw a sizable drop in Apache requests per second. A lot of frantic digging revealed that what we saw was a shift in how much bot traffic was crawling our site and not an actual drop to human interaction. This led us to build a new graph for bot traffic.
In six words or less, what are the biggest challenges your organization faces?
“There can be too much magic.” Integration of multiple systems is fantastic when it works, but when it doesn’t, getting into the weeds to figure out how that integration works can be a trying experience, especially for a new person. The keep it simple policy should rule here.
What’s the best piece of advice you’ve ever received?
“Always be coding. Always be shipping to prod.” Code is loosely used in this. It could be a chef/puppet change, or a new monitoring script, but at the end of the day, you should measure your successes for the day by solving production challenges. Also, your confidence level for releasing a change into your environment is a great representation of how streamlined your organization is. That shiny new script is useless if it never gets to production.
What principles guide your expertise in your given domain?
Don’t be afraid to make changes to production. This is heavily fostered by the idea of the Blameless Postmortem culture that Etsy has. Build confidence in what you know, support it with sane testing, and then go to production. If it breaks, fix it, learn how you prevent failure like this in the future and move on.
What is your vision for system administration in 5 years?
Automate or die. The age of hand-crafted systems is over. If you’re changing sysctl by hand to set the swappiness value, you’ve already failed. It doesn’t matter if you use chef, puppet, ansible. Automate your infrastructure and push it often.
Topics: Brainiac Corner