There's a clear correlation between a company's success and how much value it places on monitoring. If you look at industry leaders across the web, you see evidence that monitoring should not be an afterthought, and it should not be treated as a low-priority concern. The habits and policies of some of these organizations demonstrate a pattern: they invest in monitoring. They make expenditures to guarantee that their monitoring is consistent and reliable. They dedicate funds and internal resources toward those goals. It's no secret that one way to pursue success is to study those who are already successful. Recently, Twitter and Netflix have been championing some of the ideas behind their successful monitoring track records.
All businesses should take note. At VividCortex, we're familiar with the incredibly high standards that many companies now hold for observability and availability. Most organizations' databases are extremely complex. Frankly, free, simplified solutions just aren't adequate for monitoring these environments. On the other hand, a responsible investment in monitoring can ultimately provide a huge difference both in terms of ROI and by raising the standards for how that company expects its systems to function.
In a post we wrote last week, we looked at a recent article published by Twitter's Command Center team. The article explained that even 99% visibility with a minutely times series is not satisfactory for their needs. "For Twitter," the post read, "that 1% of information that minutely metrics miss is a very, very important 1% and we [need] to address this gap." Later, when discussing the shortcomings of minute-level metrics, the article allows a single sentence to act as an entire paragraph, for emphasis:
"At Twitter, we need to do better."
That idea of "doing better" is key. Twitter cares about its teams seeing and understanding their own systems. In an even more recent post, Twitter's blog explores the actual structure it uses for observability. Once again, they underline the prominence of these systems within the company: "Responsible for providing visibility into the services powering Twitter, our observability services are some of the largest scale internal systems."
Consider that: Of all the internal expenditures made by a company as huge as Twitter, observability systems are one of the biggest. Netflix has written similarly about its monitoring practices, having used deep resources to meet the complicated challenges of monitoring. Browsing their dev blog, you see topic after topic dedicated to monitoring.
What might not be obvious is that great monitoring is NOT just a luxury for massive companies with the ability to build internal solutions -- all organizations can and should monitor with scalability and high performance. The more useful takeaway from these posts is how successful companies achieve high performance, at scale, with a high emphasis placed on monitoring. Without monitoring, they may scale okay, but missing the high performance they expect -- it's too difficult to detect and solve performance problems otherwise. In other words, monitoring can be inherently expensive if it's built to provide the desired outcomes.
Inherently expensive... but worth it.