PGConf Silicon Valley was excellent this year: good organization, great speakers, and top-notch vibes in general. We were excited to see everybody visiting (and crowding!) our booth and the overall very positive community presence. Highlights included a presentation about Postgres' use at TripAdvisor, a look at Amazon RDS for Postgres, and consideration of system scalability and the future of relational database management systems.
Baron Schwartz, VividCortex's own founder and CEO, was at the Silicon Valley conference, and he's said that it was the best PGConf he's been to yet. On November 18th, Baron even led a breakout session, titled "Analyzing PostgreSQL Network Traffic with vc-pgsql-sniffer," in which he explained both the principles and practice that come together to result in VividCortex's powerful sniffer tools. Even the questions asked at Baron's talk were fantastic.
To watch a video of Baron's breakout session, find it embdedded below; you can also find more information, plus PDFs of the slides Baron used in his talk, here. And if you're interested in checking out our free Network Analyzer for PostreSQL, all of VividCortex's free tools are available on our resources page.
VividCortex's Approach to Sniffers
As a key part of his exploration of our PostgreSQL sniffer, Baron took time to examine how everything we do at VividCortex is based on a set of a few core beliefs about how databases and their contents can best be optimized and handled. These tend to be pretty broad ideas, almost philosophical in nature. Of course, when it comes to practical application of any software or tool, that magic is in the nitty-gritty, the fine tuning; but before a developer or team can get to that point, it's important for you, your company, or your system to be cognizant of some form of guiding principle.
For instance, when it comes to the ways in which we approach sniffer tools and the tasks they perform, we find it helpful to start by considering the most basic question: Why's it important to measure metrics?
Smart measuring practices can direct you toward the facets of your system where you should spend the most time to see the most impact; measurement leads to efficiency. The philosophy behind VividCortex's decisions are largely informed by the book Optimizing Oracle Performance by Cary Millsap, which provided such elegantly simple insights as “performance is response time.”
Step 1 of optimization is "measure." You can’t optimize what you can’t measure. Step 2: Have a definition of “performance” that is amenable to formal analysis.
Once you’ve figured out a way to measure queries and their response time, you’ll have a massive amount of data, and will need a way to digest and summarize that data. Group things together and sort from biggest to littlest and work down. A big part of that is determining what’s worth focusing upon: heavy hitters are key. Amdahl’s Law dictates that the improvement that can be expected by fixing a particular aspect will not exceed the percentages of resources occupied by that aspect. No matter how much faster you make it, you won’t be able to improve your situation beyond that constraint. “Don’t spend $1,000 fixing a $100 problem." So, look for the tallest tent pole. Rinse and repeat.
Use our Sniffers as a Free Resource
VividCortex is open source at heart, and, wanting to contribute back to open source, we've determined to bundle our sniffers as free products. We decided to take our core TCP sniffing and decoding libraries and libraries for the various protocols — we currently support MySQL, MongoDB, Redis, and Postgres — and bundle them into standalone executables. We reuse our core libraries plus ~20 lines of Go code. We chose a textual format that resembles the MySQL slow query log, which has many tools built for it, meaning its usable by a lot of extant tools.
What's the overhead for our sniffers? In general, at VividCortex we say that our agents consume <1% of total system CPU. Our agents also don’t consume more than 20 megabytes of memory. Taken overall, the agents are designed to be very low impact, and they do no I/O. Generally, highly loaded database servers are going to have some free CPU, and those database cycles can be consumed, essentially with zero practical performance impact. Percona did a performance benchmark of our agents running on MySQL; they confirmed that unless your system is already running at 100% CPU (an alarming and unusual scenario in itself!) you’re only going to be using resources that are already, naturally available.
Again, to check out VividCortex's free tools, they're all available on our resources page, alongside free Ebooks, Webinars, Case Studies, and more.