Blog

Published by Baron Schwartz on Aug 9, 2019 10:09:04 AM

Create Post-Mortem Reports from Notebook Templates, Export to PDF

We've released two enhancements to Notebooks that streamline collaborations to help you create post-mortem Notebooks more easily, and share them in a format that's easy for any audience to access. Both improvements are focused around how teams have told us they can get more value from Notebooks. Involving everyone in database performance is a key goal at VividCortex, so these capabilities were natural next steps for the popular Notebooks feature!

Read More
Published by VividCortex on Jul 24, 2019 12:17:13 PM

Postgres EXPLAIN Explained

ICYMI - Presentation by Baron Schwartz at PostgresConf 

We had a great time at PostgresConf 2019 in New York City! The 8th annual PostgresConf is the largest and the most diverse meeting of Postgres professionals and companies in the world. The Postgres Conference conveys the preeminent “People, Postgres and Data” experience.

Read More
Published by Baron Schwartz on Jul 5, 2019 2:29:53 PM

Configure Alerts on Deadlock Events

VividCortex can now detect deadlocks in MySQL, and generate events that you can use for alerting and notification. It's normal for an ACID transactional database to encounter deadlocks occasionally, but too many deadlocks could signal a variety of problems from SQL, to application behavior, to indexing and more. To learn more about deadlocks, lock waits, and the difference between them, read this article.

Read More
Published by Baron Schwartz on Jun 19, 2019 3:29:31 PM

Manage Database Incidents with VividCortex and Opsgenie

When you're operating always-on services—such as the databases that power production-facing interactive customer experiences—it can be stressful and hard to diagnose common problems like queries slowing down, locks piling up, or infrastructure getting overloaded. At times like these, it's important to use an incident management product and workflow to involve the right people at the right time, helping the team coordinate and communicate their workflow. And with VividCortex's new Opsgenie integration, this just got even easier.

Read More
Published by Steve Close on Jun 12, 2019 6:30:00 PM

VividCortex Achieves SOC 2 Type II Certification

We are pleased to announce that VividCortex has successfully completed SOC 2 Type II certification. This reaffirms and advances our ongoing commitment to protecting our customers—and by extension, their customers. The report is the result of another intensive effort by our dedicated security team, providing evidence to our auditors so they could examine our processes and controls. The audit was completely successful, as with our first audit, and was a clean and uneventful process.

Read More
Published by VividCortex on Jun 8, 2019 10:21:21 PM

Percona Live 2019 Recap

Optimizing Database Performance, DevOps, the Changing Face of Open Source, and Customer Appreciation 

We had a great time at Percona Live 2019 in Austin, TX - delivering keynotes and spending quality time with our great customer base. Below you'll find recaps of our experiences! It was great to see you; relive it with us! 

Read More
Published by Baron Schwartz on Jun 6, 2019 11:11:48 AM

See App Releases Visually with Change Markers

With VividCortex's deploy markers, you can quickly and visually identify correlations between releases in your app or infrastructure, and changes in system behavior. It's a one-liner to configure your deploy tooling to send an event to VividCortex when you launch changes. After that, you can select them from the navigation bar. Deploy events appear as vertical markers on charts and graphs everywhere in the app.

Read More
Published by Racquel Yerbury on May 22, 2019 1:50:28 PM

Taming the Database: How Observability Spurs Continuous Delivery

For tech professionals it’s intuitive that higher software delivery and operational performance — SDO performance is a new-ish moniker — yields higher organizational performance, or at least substantially contributes to it.  But now, five years of research from the DevOps Research and Assessment (DORA) team have consistently provided this as a data-driven conclusion for business leaders too. And when it comes to the specific technical practices that improve SDO performance, rising decisively to the top is a core DevOps workflow: continuous delivery—discovering what positively impacts it, honing those catalysts, and optimizing.

Read More
Published by VividCortex on Apr 9, 2019 8:00:52 PM

Essential Capabilities for Applying DevOps to Your Database

Software industry leaders have been talking about DevOps for more than a decade. Most top firms say that they’re committed to a holistic DevOps philosophy and that they’re trying to put it to work across their organization, integrating roles and functions throughout the development pipeline. Yet many of them overlook the one area where problems are most common and consequential: the database.

Read More
Published by VividCortex on Mar 8, 2019 11:11:50 AM

How to Architect and Build Highly Observable Systems

We’ve completely revised one of our most popular free ebooks, How to Architect and Build Highly Observable Systems. This book, originally written in 2015, has been downloaded by thousands of visitors and has generated extensive discussion. The updated 2019 edition incorporates many of the great points our readers have raised, as well as reflecting the fast-evolving state of the observability community and industry, which is much different now than it was a few years ago.

Read More
Published by Jordan Raine on Feb 17, 2019 4:09:05 PM

Using FORCE INDEX: When Your Query Optimizer Gets It Wrong

A Guest Post from Jordan Raine, Clio

Jordan Raine is a staff software developer at Clio, the leading provider of legal practice management, client relationship management, and client intake software. Clio is dedicated to helping lawyers meet client needs and run efficient, profitable practices. Jordan wrote this post to help mySQL developers save time/effort and ultimately improve their customers' experience. 

Read More
Published by Baron Schwartz on Feb 15, 2019 11:22:15 AM

Databases, Shift-Left, and Empathy: What You Missed at DevOpsDays Charlotte

Last week several people from VividCortex traveled to attend DevOpsDays Charlotte, where I gave the opening keynote, we had a table to demonstrate our product, we caught up with friends and customers we haven't seen in a while, and last but certainly not least we shared in lots of DevOps conversations! Here's a recap of some highlights I enjoyed at the conference.

Read More
Published by Baron Schwartz on Feb 8, 2019 2:45:23 PM

Inspecting Index Usage In MySQL, PostgreSQL, and MongoDB

In my recent post about the Left-Prefix Index Rule, I explained how queries can use all or part of a compound (multi-column) index. Knowing what makes an index fully usable by a query is important, but it's also important to know how to figure out how much of an index a query is able to use. In this article I'll show you how to do this by reading the query's explain plan. This article covers MySQL, Postgres, and MongoDB.

Read More
Published by Baron Schwartz on Feb 5, 2019 11:29:13 AM

The Left-Prefix Index Rule

There's an important heuristic in database indexing that I call the left-prefix rule. It helps you understand whether the database will be able to make the best use of a multi-column index to speed up a query. It's one of a small handful of really important things to understand about indexes!


The left-prefix rule can be confusing, but once you grok it, it's simple and easy to use. It applies to MySQL, PostgreSQL, MongoDB, and many other databases. In this post I'll explain what the left-prefix rule means and how it works.

Read More
Published by Baron Schwartz on Feb 1, 2019 11:04:00 AM

Covering Indexes in MySQL, PostgreSQL, and MongoDB

A covering index is a fantastic query performance optimization. An index covers a query when the index has all the data needed to execute the query, so the server can retrieve the query’s data without reading any rows or documents. Covered queries are usually SELECT queries, but in addition to reads, indexes can cover portions of what a write query needs to do its work.

In this article I’ll explain what a covering index is in a bit more detail; the technical implementation and caveats in MySQL, PostgreSQL, and MongoDB; and how to check that a query is using a covering index.

Read More
Published by Baron Schwartz on Jan 28, 2019 2:36:00 PM

What is Concurrency in a Database?

Databases are confusing! Not only are they complicated bits of software, but database jargon is really complicated. Part of this comes from the number of words that have multiple meanings: depending on which community you spend time in, there are multiple meanings for words like schema, cluster, and even database.

Read More
Published by Baron Schwartz on Jul 30, 2018 1:47:27 PM

What is Cardinality in Monitoring?

I wrote a couple of “definitions and nuances” posts about terminology in databases recently (cardinality, selectivity), and today I want to write one about cardinality in monitoring, as opposed to cardinality in databases. If you’ve seen discussions of “high-cardinality dimensions” or “observability requires support for high-cardinality fields” this is what we’re talking about today. So what does it mean?

Read More
Published by Samantha Toet on Jul 10, 2018 12:43:04 PM

VividCortex Welcomes Amena Ali as CEO

Exciting news: today we announced the appointment of Amena Ali as Chief Executive Officer and member of the Board of Directors. Company founder Baron Schwartz will now assume the role of Chief Technology Officer. “I am proud of the product and culture we’ve built,” said Baron. “I’ve always believed that the keys to a rewarding career are being challenged, making a meaningful impact, and working with delightful people. Amena joins at the perfect time to scale the company and team, while I redouble my focus on our customers and the product vision for the next wave of monitoring needs.”

Read More
Published by Baron Schwartz on Jul 6, 2018 12:11:30 PM

What is Cardinality in a Database?

Databases have a lot of jargon, and cardinality is one of those words that experienced people tend to forget that they didn’t know once upon a time. But if you don’t know it—and it takes a while to really get comfortable with cardinality—it’s super confusing when the DBA just drops it into the middle of a sentence without slowing down! Fear not: I’ve got you, as they say.

Read More
Published by Samantha Toet on Jun 28, 2018 1:24:31 PM

How NOT to Monitor Your Database

Do you have experience putting out backend database fires? What were some things you wished you had done differently? Proactive database monitoring is more cost efficient, manageable, and sanity-saving than reactive monitoring. We reviewed some of the most common mistakes - too many log messages, metric “melting pots,” retroactive changes, incomplete visibility, undefined KPIs - and put together an action plan on how to prevent them. From our experience, we've listed out the top 5 biggest (and preventable!) database monitoring pitfalls.

Read More
Published by Samantha Toet on Jun 19, 2018 10:33:33 AM

VividCortex & Grafana: Better Together

Many of our customers have asked us to integrate with Grafana, the popular open platform for analytics and monitoring. The wait is over: we’ve released a plugin that lets you retrieve and display VividCortex metrics in your own Grafana dashboard. With this integration, you can combine metrics and dashboards from all of your systems and tools into a single source of truth that your team is already familiar with, giving them the ability to explore and use VividCortex’s data without having to learn a new tool.

Read More
Published by Samantha Toet on Jun 1, 2018 4:26:17 PM

Monitor Critical Databases Confidently with the Sensitive Data Vault

Building extremely deep monitoring as a SaaS product has a drawback: we capture too much data for some customers’ compliance requirements. As a result, some companies have been unable to deploy us, or have had to redact data before sending it to our cloud platform. To address this, we built the Sensitive Data Vault, a highly secure, completely on-premises storage module for the most critically private data that must never leave the customer’s firewall.

 

Read More
Published by Samantha Toet on May 11, 2018 5:17:30 PM

How SendGrid Ships Better Code Faster with VividCortex

VividCortex CEO Baron Schwartz and SendGrid DBA Silvia Botros teamed up to discuss how performance monitoring leads to better, faster code deployment. This 30-minute webinar covers: 

  • How developers can deploy code faster and more safely.
  • A close-up view of a health- and monitoring-focused work environment.
  • How database monitoring fits into a culture of DevOps and lean, agile development. 

Read More
Published by Baron Schwartz on Apr 26, 2018 12:37:35 PM

The High Cost and Low Benefit of Unused Index Advice

You have to be careful what you wish for. The signal:noise ratio of finding out which indexes are “unused” is low. This isn’t obvious at first glance, but upon deeper inspection, there’s often not much to be gained, and it can even harm you. Let’s look at why.

 

Read More
Published by Andrew Morgan on Mar 30, 2018 11:15:00 AM

What's New: Product Updates  

We’ve added several major features since our last product update. These capabilities help you understand your database’s performance and health, get expert insights about topics like configuration and security, explore individual sample query executions fluidly, and store your most sensitive data on-premises instead of in our cloud platform. If you have any questions, contact us!

Read More
Published by Preetam Jinka on Mar 20, 2018 9:05:45 AM

Holt-Winters Forecasting Simplified

Holt-Winters forecasting is a way to model and predict the behavior of a sequence of values over time—a time series. Holt-Winters is one of the most popular forecasting techniques for time series. It’s decades old, but it’s still ubiquitous in many applications, including monitoring, where it’s used for purposes such as anomaly detection and capacity planning.

Unfortunately, Holt-Winters forecasting is confusing, so it’s often poorly understood. We want to fix that, so we wrote this post: a visual introduction to Holt-Winters.  

Read More
Published by Aisha Robinson on Mar 15, 2018 8:30:00 AM

Strata Conference 2018: What You Missed

We just got back from the west coast where our own CEO Baron Schwartz presented at Strata Data Conference 2018.

Baron held a session, Why Nobody Cares About Your Anomaly Detection at the event and in case you missed it we wanted to make sure we passed it along! In the session, Baron shares why he now sees anomaly detection as a very limited tool, to be used for specific purposes and with careful attention to design and context (including culture) and only to be considered as a part of an overall solution—not a solution itself.  View the entire presentation below.

 

Read More
Published by Preetam Jinka on Mar 12, 2018 2:27:00 PM

Numeric Optimization in Golang with Nelder-Mead and Gonum

Numeric optimization is a set of techniques for fitting a function to a dataset by iteratively refining estimates. Numeric optimization is one way to do things like curve-fitting and parametric estimations. We use it in some time series forecasting techniques we implemented recently, where the forecasting models use parameters such as seasonality and trend that we can’t determine in advance. We use numeric optimization to select the optimal parameters based on the specific dataset we’re trying to forecast.

The gonum package https://github.com/gonum/gonum is a great set of numeric libraries for Go. It has a numerical optimization package. In this post I’ll show how to fit a line to 2D points with the optimize package. This is the simplest example of how to do numerical optimization with the package. It’s a good illustration of how to set up an optimization, and helps show how to get started so you can do more complicated things if you want.

Read More
Published by Aisha Robinson on Mar 7, 2018 10:33:27 AM

Did You Miss Us at DevOps Days Charlotte?

DevOps Days Charlotte 2018 was a great event and we can't wait to return next year! Over 200 development, infrastructure, operations, information security, management and leadership professionals gathered to discuss the culture, processes and tools to enable better organizations and innovative products.

VividCortex engineer Preetam Jinka held a session, DevOps Comes To The Database at the event and in case you missed it we wanted to make sure we passed it along! In the session, Preetam discussed the importance of your database and how while databases are stateful they are changing and becoming more like applications. View the entire presentation below.

 

Read More
Published by Matt Culmone on Mar 1, 2018 9:21:04 AM

Announcement: p99 Percentile Metrics

I’m pleased to announce that VividCortex now offers 99th percentile metrics to help understand latency outliers in a query workload. These metrics provide visibility beyond the average latency, help identify the worst outliers, and improve focus on the customer experience. They are offered for all of the databases we currently support when using On-Host monitoring.

Read More
Published by Aisha Robinson on Feb 26, 2018 3:40:16 PM

How To Buy Software 2018

Buying software solutions for your company is a skill just like any other business practice. To be effective at purchasing it is necessary to have a clear understanding of what you require and a plan on how to evaluate potential vendors. When time is also a factor a working knowledge of the internal buying processes and expectations of your employer are also important.

Read More
Published by Aisha Robinson on Feb 20, 2018 9:23:50 AM

Come See VividCortex at DevOps Days Charlotte!

We're thrilled to be kicking off 2018 at DevOps Days Charlotte! DevOpsDays Charlotte will bring 200+ development, infrastructure, operations, information security, management and leadership professionals together to discuss the culture, processes and tools to enable better organizations and innovative products.

Read More
Published by Aisha Robinson on Feb 15, 2018 3:39:14 PM

See You on the Road at 2018's Shows and Events!

Calling all developers, DBAs, engineers, SREs, and database aficionados: we’re hitting the road and hope to see you along the way. It's always a highlight of our year when we get to meet up with our users and friends face-to-face, and we'll be attending events across the country (and abroad!) for the rest of 2018. If you see our booth, come say hello. 

Read More
Published by Matt Culmone on Jan 23, 2018 9:07:24 AM

It's Inevitable. Your Database Will Fail

Will my database fail?

Databases fail. No one can promise 100% uptime, it’s impossible. Whether the database is large, small, on-premise or cloud-based, all have the potential to fail. This could be transactional errors, system crashes, out of memory errors or out of disk space errors. Sometimes they fail suddenly and sometimes they just can’t cope with growing demand and “fail slowly” over a period of time.

Read More
Published by Preetam Jinka on Jan 17, 2018 9:16:35 AM

Slow Queries? Move Fast to Fix Them

What are the effects of slow queries?

Slow queries can have serious negative impacts on your business. The impacts will vary depending on your architecture, industry, or application but the bottom line is to not ignore them. Let's discuss whom and what they will impact.

Read More
Published by Baron Schwartz on Jan 4, 2018 3:47:50 PM

2017 Year in Review at VividCortex

It’s easy to observe (pun intended) in the rear-view mirror. Hindsight bias aside, 2017 was a big year for VividCortex and our customers! We shipped lots of features and made tons of progress on the business. Here’s a brief overview of some of our proudest moments from 2017.

Read More
Published by Steve Close on Dec 13, 2017 3:59:24 PM

Committed To Security: VividCortex and SOC 2 Type 1

Security has always been a priority at VividCortex, and we have architected and built our product for security from the beginning, frequently conducting exercises such as third-party penetration tests and code reviews. Today we are pleased to announce that VividCortex has successfully completed SOC 2 Type I certification.  The report comes after an intensive (and completely successful) auditing process, and is tangible and transparent proof of our commitment to customer protection.

SOC 2 compliance is neither an easy process nor a useless one. Most of us at VividCortex have worked in companies that were subject to various types of security requirements. What we like most about SOC 2 is that it’s sensible and legitimate: the requirements are both common sense and rigorous types of things you must do to actually be secure. SOC 2 Type 1 attests that the VividCortex controls were designed and implemented to meet the criteria for Security, Availability, Processing Integrity and Confidentiality.

Read More
Published by Baron Schwartz on Dec 3, 2017 3:51:06 PM

Monitoring, Analytics, Diagnostics, Observability, and Root Cause Analysis

Monitoring is a hopelessly overloaded term in tech culture. The term now carries decades of inaccurate and imprecise use. The result is that several people can be engaged in an earnest conversation about monitoring and, despite efforts to get each other to see what they mean, remain on totally different wavelengths. I know, because I’ve seen it happen many times. It’s amazing how many times I’ve seen people frustrated with each other because they mean different things when talking about these words.

Read More
Published by Baron Schwartz on Nov 7, 2017 5:09:42 PM

Hierarchical Observability with RED

I've written before about the minimal set of metrics that can serve effectively as application/service vital signs. One such set is the RED acronym, which stands for Request Rate, Request Errors, and Request Duration. (I'll write in the future about what's missing from this acronym, but it'll serve the purpose for now).

Read More
Published by Baron Schwartz on Oct 13, 2017 10:16:40 AM

Monitor Your Citus Cluster With VividCortex

VividCortex now has support for monitoring Citus, a scale-out solution for massively sharded data storage and processing using standard open-source PostgreSQL. Citus (previously known as CitusDB) layers an additional set of capabilities on top of PostgreSQL. The result is a powerful distributed database that shards your data and lets you ingest a firehose of data, while running complex real-time analytics queries in parallel across nodes for blazing fast performance.

Read More
Published by Baron Schwartz on Oct 6, 2017 12:20:56 PM

Explode Charts To Drill In With VividCortex

VividCortex is designed with two key scalability requirements in mind: it must perform well in large environments with many hosts, and the UI must help users understand and inspect those environments easily. The universal time selector, the host filter, and other features let you start with a top-level summary, grasp what’s happening in your entire environment no matter how many hosts, and then drill into hosts, time ranges, queries, and metrics of interest. We’ve enhanced our charting and graphing capabilities to make this same zoom-in-drill-down inspectability easier, too.

Read More
Published by Baron Schwartz on Oct 5, 2017 4:02:17 PM

Monitoring and Observability with USE and RED

Modern systems can emit thousands or millions of metrics, and modern monitoring tools can collect them all. Faced with such an abundance of data, it can be very difficult to know where to start looking when you’re trying to diagnose a problem. And when you’re not in diagnosis mode, but you just want to know whether there’s a problem at all, you might have the same difficulty. What are the truly key KPIs coming from your systems?

 

Read More
Published by Baron Schwartz on Sep 26, 2017 11:01:16 PM

Custom Dashboards in VividCortex

VividCortex now offers custom dashboards, which are collections of charts you can curate as you wish. Custom dashboards are shared across your team, so they're a great way to ensure everyone has access to the metrics that you use to monitor and inspect system status and health. And you can give them meaningful names, which is one of only two hard things in computer science, so your dashboards can be full of win.

Read More
Published by Baron Schwartz on Sep 21, 2017 4:31:49 PM

Find Queries That Need Indexes With VividCortex

Queries can be slow or resource-heavy for many reasons, but one of the most common is that the table doesn't have a good index for the query. This is true for MySQL, PostgreSQL, and especially for MongoDB. Adding an index where it's missing is often a night-and-day improvement, as shown in this screenshot of a query's performance. The index was added midway through the time range, and the query got dramatically faster:

Read More
Published by Baron Schwartz on Sep 14, 2017 4:56:32 PM

Monitoring Isn't Observability

Observability is all the rage, an emerging term that’s trending up very quickly in certain circles even while it remains unknown in others. As such, there isn’t a single widely understood meaning for the term, and much confusion is inevitably following. What is observability? What does it mean? Perhaps just as importantly, what is observability NOT?

TL:DR; Monitoring tells you whether a system is working, observability lets you ask why it isn't working. Tweet this

Read More
Published by John Potocny, Ewen Fortune, and Alex Slotnick on Sep 6, 2017 11:02:05 AM

The Basics of Sharding in MongoDB

Sharding is a mechanism widely used in today's most popular database systems, an effective way to divide, categorize, and organize data into manageable chunks. In some database types, such as MySQL, the use of sharding is an architectural decision, while in other databases, such as MongoDB, sharding is a fully supported, native feature. If effectively used, it can play a role in scaling a system effectively to meet organizational or business demands.

Read More
Published by Baron Schwartz on Aug 18, 2017 6:20:46 PM

Meet The New DBA, Different From The Old

There’s a rapid shift taking place in today's technology organizations: the role of the DBA is being redefined and increasingly replaced by other roles and specialties. This is happening even as data explodes—in fact, it's happening precisely because data is exploding. It's a trend that's not only moving quickly, it's accelerating, and it sometimes takes people by surprise.

Read More
Published by John Potocny on Aug 17, 2017 4:28:40 PM

The 5 Principles that a Performance Management Service Should Follow

I’ve recently taken some time to think about what it means for our organization to operate and identify as a "performance management service." It's not as simple as placing ourselves in a specific industry category or engineering a certain kind of app. There are other, complex, less tangible elements too: how we interact with the systems we monitor, our responsibilities as experts in our field, the obligations we have to our customers, and the standards we hold ourselves to.

Read More
Published by Alex Slotnick on Aug 4, 2017 9:53:09 AM

Throughput Is the One Server Metric to Rule Them All

If you've ever been responsible for maintaining the performance of a busy app or website, there's a good chance the task often felt complex, with far too many metrics. Since you can measure so much in a modern system, how do you tell what's really important? In reality, though, as complicated as your job might feel, your real concerns can actually be understood through a much simpler — and more manageable — single metric: throughput. For service providers, server performance is all about resource management.

Read More
Published by Baron Schwartz on Jul 31, 2017 10:17:00 AM

How We Encrypt Data In MySQL With Go

A SaaS product needs to use security measures you might not ordinarily use in an on-premises solution. In particular, it’s important that all sensitive data be secured. Encryption plays an important role in information security. At VividCortex, we encrypt data in-flight and at-rest, so your sensitive data is never exposed.

Read More
Published by Alex Slotnick on Jul 25, 2017 1:32:08 PM

Brainwaves Podcast Ep 5: An Interview with Bruce Momjian

In this episode of Brainwaves, Alex chats with Bruce Momjian, a member of PostgreSQL's core development team, at the PGConf Local meetup in Philadelphia, July 2017. They discuss Bruce's background with PostgreSQL, some of the history of PostgreSQL’s development, how community support helps drive the database's evolution, and the importance of PostgreSQL's hybrid userbase.

Read More
Published by John Potocny and Ewen Fortune on Jul 24, 2017 12:31:35 PM

Deciding Which Storage Engine is Right for You: WiredTiger

We've recently written several articles about the history and advantages of MongoDB's storage engines, explaining why WiredTiger replaced MMAPv1 as the default engine and the contexts in which MMAPv1 might still be the best choice for some users.

In this article, we'll continue these conversations and consider how you should decide which storage engine is right for you, this time returning to WiredTiger. It may be MongoDB's default engine, but is it the best option in every situation?

Read More
Published by John Potocny and Ewen Fortune on Jul 19, 2017 5:17:49 PM

You QA Most of Your System — What About Your Database?

For virtually all development teams, testing code is a given: It's one of the most important parts of software development. Whether your organization includes a separate team devoted to QA, or your developers are testing their own code, QA is the primary way your team ensures that your application's logic is working correctly, and it's the best way for you to identify issues as early as possible.

As a result, QA is critical for engineering velocity, and it helps shape your users' overall experience when engaging with your product. Nobody likes finding a broken app or website. But what about quality assurance for a database? Do most teams apply the same QA practices to improve their data tier? Do many teams even know how to perform database QA? In this article, we'll talk about how you and your team can apply the same high standards of QA principles to your database, which many organizations often overlook.

Read More
Published by Alex Slotnick on Jul 18, 2017 4:07:38 PM

PGConf Local in Philly: PostgreSQL Performance, in Theory and Practice

A few members of the VividCortex team had the chance to attend PGConf's local meetup in Philadelphia last week. The group came together to discuss anything and everything Postgres, including new features to be included in PostgreSQL 10, building a scalable time-series database on top of PostgreSQL, and common mistakes people make when migrating from Oracle to PostgreSQL.

Read More
Published by Alex Slotnick on Jul 7, 2017 4:14:35 PM

Why Tech Ops Needs to Care About User Experience

If you work on an app’s backend, you’re probably used to looking at server-side metrics like utilization and backlog. Alas, this can give you major tunnel vision, because it can easily cause you to ignore the user’s experience. Why does this empathy matter?

Read More
Published by John Potocny and Ewen Fortune on Jul 6, 2017 7:43:12 PM

What is MongoDB's _id Field and How to Use It

MongoDB's users are well known for valuing the database's capacity for growth. After all, the system's tagline is "for giant ideas," and for good reason.  The system supports extreme scalability and has been designed to store exceptionally large numbers of documents. Among all the data that a MongoDB system handles, however, there's only one field that exists across every document: the _id field.

Read More
Published by Alex Slotnick on Jun 30, 2017 5:20:51 PM

You Need to Consider Two Perspectives to Understand System Performance

There are two fundamental points of view for thinking about system performance. Most people tend to experience a system from only one side, and they can often analyze that perspective extremely well. This makes sense—most people are in a position where only one point of view occurs naturally. But that doesn't mean you shouldn't try to see both perspectives. Each one is equally valid, and without a complete view, you have no hope of understanding a definition of your system's performance as a whole.

Read More
Published by John Potocny and Ewen Fortune on Jun 23, 2017 6:08:47 PM

Deciding Which Storage Engine is Right for You: MMAPv1

In a recent article, we began an examination of MongoDB storage engines by explaining why WiredTiger is the current default engine. In this piece, we'll continue discussing storage engines and start to look at the specific characteristics of other available options, beginning with MMAPv1, to help readers understand which one is right for their system and workloads.

Read More
Published by Alex Slotnick on Jun 23, 2017 3:44:22 PM

MongoDB World: "MongoDB Performance in Theory and Practice"

We had a fantastic time this week in Chicago at MongoDB World, where VividCortex participated as a sponsor and we unveiled new updates in our product's MongoDB support. Baron Schwartz, VividCortex's founder and CEO, also presented a talk, titled "MongoDB Performance in Theory and Practice." 

Read More
Published by Preetam Jinka on Jun 22, 2017 4:45:00 PM

Exponential Smoothing for Time Series Forecasting

Time series anomaly detection is a complicated problem with plenty of practical methods. It’s easy to find yourself getting lost in all of the topics it encompasses. Learning them is certainly an issue, but implementing them is often more complicated. A key element of anomaly detection is forecasting - taking what you know about a time series, either based on a model or its history, and making decisions about values that arrive later.

Read More
Published by John Potocny and Ewen Fortune on Jun 19, 2017 5:51:49 PM

When to Use explain() in MongoDB

In MongoDB, the command db.collection.explain() is an important operation that can help you understand exactly how your queries execute and behave. Once you have a grasp of how MongoDB handles explain(), using it lets you examine queries in a number of ways.

Read More
Published by John Potocny and Ewen Fortune on Jun 14, 2017 11:43:05 AM

Why WiredTiger is the Default MongoDB Storage Engine

Last week, when we published our blog post about working-sets in MongoDB, one of our colleagues told us that he'd been aware that MMAPv1 had previously been MongoDB's default storage engine, but he didn't know why WiredTiger had since taken its place. We saw this as an opportunity to write a follow-up post to explain some of the contrasts between the two and explore the advantages of WiredTiger, which (we think) led MongoDB to making it the system's default storage engine.

Read More
Published by Alex Slotnick on May 19, 2017 3:24:11 PM

Congratulations to Baron Schwartz, CBIC's 2017 Entrepreneur of the Year

We're excited to share that last night, Baron Schwartz—VividCortex's very own, very deserving founder and CEO—was recognized as "Entrepreneur of the Year" by the Charlottesville Business Innovation Council (CBIC) at their 19th annual CBIC Awards. The award honors a leader who "embodies all the elements of successful tech-centric entrepreneurship, including innovation, employment, leadership, finance, determination, and persistence."

Read More
Published by Alex Slotnick on May 4, 2017 4:48:26 PM

Highlights from Percona Live 2017

Percona Live 2017 is a wrap! What did we think? Baron Schwartz shares a few of his thoughts in the highlight reel below. Some of our favorite parts of the show included:

  • This year there was an especially great combination and blend in how the show presented different open-source database technologies. The conference did a great job bringing the ever-expanding industry together.
  • MongoDB showed up in a big way. MongoDB is emerging as a true leader among NoSQL tech. 
  • Percona XtraDB Cluster showed its true capabilities, deployed large-scale, in production. It's demonstrating success handling heavy workloads. 
  • Time series data asserted itself as a new category with special prominence. Several vendors at the show addressed this category specifically, alongside a few new open-source projects. 
  • The show offered an entire track on AWS, with much depth and a variety of approaches.

Read More
Published by Alex Slotnick on May 4, 2017 10:16:57 AM

Percona Live 2017 Replay: Baron Schwartz's Keynote Address

Last week, Percona Live 2017 gave us the chance to meet with our friends and peers at the best open-source database conference in the world. Hundreds of thought leaders and experts came together to discuss cutting edge ideas in our industry, demonstrating how rapidly monitoring solutions and data technologies continue to evolve.

Read More
Published by Alex Slotnick on Apr 24, 2017 2:59:19 PM

Come See VividCortex at Percona Live This Week!

We're thrilled it's finally that time of year: Percona Live 2017! This event is among the most important in the database communitythere's no other conference like it, where you can find such a meeting of industry leaders, gathered to discuss open-source databases and monitoring.

Read More
Published by Baron Schwartz on Apr 14, 2017 2:18:21 PM

Correlated Metrics in Monitoring

If you work with monitoring or monitoring tools much, you’ve probably seen the phrase “correlating” here and there. For example, monitoring vendors often say you can use their product to correlate metrics. Issue 157 in the popular Prometheus monitoring system’s GitHub repository is to add support to correlate multiple metrics in the same graph.

I’ve noticed that when monitoring-related discussions mention correlation, the meaning is usually pretty vague. It often seems to refer to just graphing multiple things on a single chart with the same timescale.

Read More
Published by Alex Slotnick on Apr 12, 2017 4:17:59 PM

How to Replace a Wing in Midair, and Other Development Safety Tips

In this episode of Brainwaves, Baron Schwartz and Jay Ennis -- VividCortex's CEO and VivdCortex's VP of Product Development -- join Alex to discuss some of the concepts that help engineering teams operate and build safely. Jay and Baron talk about what they aim to protect and avoid when setting up safety protocols for a team, and they discuss why some commonly held beliefs, like root cause analysis, are traps. Ideas from other industries -- such as aviation, EMT emergency reporting, and horse rearing -- lend helpful perspectives and models.

Read More
Published by Alex Slotnick on Apr 10, 2017 12:54:18 PM

Five Events and Presentations We're Excited to See at Percona Live 2017

Percona Live 2017 is only a couple weeks away, and we're making our list of can't-miss events that we plan to catch while we're there. If you ask us, VividCortex's Baron Schwartz and Preetam Jinka's talks and breakout sessions are obvious picks, but we're just as excited to catch some of the fantastic presentations by our friends and peers. Need help getting your schedule in order before Percona Live kicks off? Below are five agenda items that we're particularly excited to see in Santa Clara.

Read More
Published by Ewen Fortune on Mar 31, 2017 1:06:08 PM

Bonanza Cuts Load in Half with VividCortex

Working with our users at Bonanza earlier this week, we saw their team demonstrate a great example of how monitoring insights can lead to a relatively simple — but impactful —  MySQL system tweak. In this case, the adjustment Bonanza made resulted in huge improvements to their total query time.

Read More
Published by Alex Slotnick on Mar 24, 2017 12:02:32 PM

What's Really Behind Your Database Bottleneck? - App Developer Magazine

Database management, monitoring, and general health call for a range of specialized resources. To ensure that databases function properly, you need specific monitoring, you need focused knowledge of how databases work, you need to understand what makes the database a unique layer of your technology stack. However, the database doesn't just matter to database experts and DBAs -- it plays a major role for entire organizations.

Read More
Published by Alex Slotnick on Mar 23, 2017 12:15:00 PM

Common Pitfalls When Using database/sql in Go

Here at VividCortex, we’re huge fans of the Go language and its database access library, database/sql. As you’ve probably seen firsthand, the surface area of database/sql is pretty small, but there’s a lot you can do with it. That includes plenty of risk for error and deceptive mistakes. This blog post is dedicated to some of the past mistakes we’ve made ourselves, in hopes that you won’t also make them when the time comes.

Read More
Published by Kelsey Uebelhor on Mar 16, 2017 5:57:56 PM

See You on the Road at 2017's Shows and Events!

Calling all developers, DBAs, engineers, SREs, and database aficionados: we’re hitting the road and hope to see you along the way. It's always a highlight of our year when we get to meet up with our users and friends face-to-face, and we'll be attending events across the country (and abroad!) for the rest of 2017. If you see our booth, come say hello. 

Read More
Published by Alex Slotnick on Mar 2, 2017 4:47:45 PM

Chaos and the Fugly Five

Maintaining a SaaS service in the cloud inherently involves nonstop complexity, inevitable fixes, and the threat of issues — especially when that SaaS service involves consistent growth and active changes. In order for VividCortex's product team to reliably pinpoint and head off potential problems, we've needed to use creative and resourceful DevOps systems. In this post, we'll be discussing one of the methods we use to keep potential problems in check: "Chaos and the Fugly Five." (Its name might not communicate the full seriousness of its use, but the entire extent of its creativity no doubt comes through.)

Read More
Published by Alex Slotnick on Feb 23, 2017 6:13:39 PM

First Episode of New "Brainwaves" Podcast Series - Listen Now

We're excited to share the first episode of VividCortex's new podcast, Brainwaves! Join us as we dig into topics related to databases, data management, technology startups, and engineering in general. This show is an opportunity for us to get our synapses firing about the cutting edge developments in tech—databases in particular.

Read More
Published by Alex Slotnick on Feb 17, 2017 2:25:15 PM

How Online Florists Show their Databases Love before Valentine's Day

Baron Schwartz's recent article on DZone, "How Online Florists Can Show Their Databases Some Love Before Valentine's Day," helped flower sellers across the country prepare for one of their biggest shopping events of the year. Like other retail industries, florists need their online systems to maintain high-performance through heavy traffic periods, so outages or slow systems don't interrupt shoppers' activities. This requires advanced preparation for the database. Capacity planning, code slushes, and being able to expect the unexpected are all key.

Read More
Published by Alex Slotnick on Feb 11, 2017 7:42:00 PM

Shopify Eliminates High Latency Queries From Redis and MySQL

As intuitive and streamlined as ecommerce technology might seem from the user's perspective, it involves so much data that engineering ingenuity and smart database management must constantly deliver in order to keep up. At organizations like Shopify—responsible for the easy and reliable transactions at top brands around the world—that excellence of performance involves deep monitoring of their MySQL core and their Redis caching infrastructure, plus insightful query profiling, packet captures, and the admittance of developers to platforms that measure database performance.

Read More
Published by Alex Slotnick on Feb 11, 2017 6:38:00 PM

Optimizing VividCortex App Downloads Through Code Reduction

We've recently been able to achieve new optimization for VividCortex app downloads by making the code smaller, thanks to several successful minification and reduction projects by our frontend team. Of course, minification is an automated process, which we've been performing since day one. In addition, we recently performed an exhaustive revision of our own code, in order to find and remove any unneeded aspects.

Read More
Published by Baron Schwartz on Feb 10, 2017 4:42:00 PM

JSON Support in PostgreSQL, MySQL, MongoDB, and SQL Server

If you've been watching the evolution of database technologies over the past few years, you've seen how quickly JSON has quickly cemented its position in major database servers. Due to its use in the web front-end, JSON has overtaken XML in APIs, and it’s spread through all the layers in the stack one step at a time.

Read More
Published by Alex Slotnick on Jan 31, 2017 6:01:57 PM

VividCortex Newsletter: January 2017

The January 2017 edition of the VividCortex newsletter includes a video case study with Shopify; our ebook "The Hidden Costs of Data Engineering"; a recording of the webinar "How to Be a Performance-Driven Engineer"; a Case Study with SalesLoft; and more.

Read More
Published by Preetam Jinka on Jan 27, 2017 3:20:50 PM

Three Things That Differentiate Amazon Aurora From MySQL

It's not always obvious what makes one database type distinct from another. What are the most significant ways that Amazon Aurora is different from MySQL? Clear separators aren't always featured or widely known, but even slight variables between two databases can prove valuable in choosing which one is right for you and your organization.

In the case of Aurora, there are at least three interesting things that make it unique and that present opportunities for particular uses. (Thanks in advance to @saileshkrish for helping us stay in-the-know on what Aurora can do.)

Read More
Published by Alex Slotnick on Jan 24, 2017 6:51:13 PM

How Dyn Reduces Time To Resolution

The Internet is a massive labyrinth of connections and intersections. Without a map or a guide, it would be almost impossible to find anything inside of it. That's why we need a translator to help us understand DNS, the "Internet's phone book." For many organizations, websites, and companies, that translator is Dyn, the market's leading Internet Performance Management (IPM) solution.

Read More
Published by Alex Slotnick on Jan 19, 2017 4:43:46 PM

5 Tips for DBAs on Working Remotely

For many professionals who've never had the chance to try it, full-time remote work might seem like pure upsideeven luxurious. On days when rush hour commutes, shared fridge space, and raucous office mates threaten to overwhelm, who wouldn't prefer the option of settling into a spacious, at-home desk, with the family dog snoozing at one's heels?

But like any logistical detail of a job, working remotely comes with its own unexpected challenges. This is especially true for people whose roles are highly variable, sensitive, and central to an organization's ability to function, such as database administrators'.

Read More
Published by Baron Schwartz on Jan 11, 2017 10:44:00 AM

MySQL Query Performance Statistics In The Performance Schema

Quite some time ago, when we added the ability to monitor MySQL query performance statistics from MySQL’s PERFORMANCE_SCHEMA, we learned a number of valuable lessons, still worth sharing. There are definitely right and wrong ways to do it. If you are looking to the P_S tables for monitoring MySQL query performance, this blog post might save you some time and mistakes.

Read More
Published by Alex Slotnick on Dec 31, 2016 10:58:49 AM

2016: VividCortex's Exciting Year in Review

As New Year's festivities start up around the world, VividCortex has many reasons to join the celebration. We're exhilarated and grateful to think back on how far we've come over the past 12 months. This has been a year of non-stop progress for our team and in what we've seen VividCortex's users accomplish.

Read More
Published by Alex Slotnick on Dec 22, 2016 4:13:19 PM

Mystery: Constant Query Time Despite Varying Loads

A little while awhile ago VividCortex's engineering team encountered a minor mystery when observing some trends in time_us metrics in a MongoDB database. While the conclusion of our investigation was a kind of combination of "D'oh!" and "Ah ha!," we thought our considerations along the way were interesting and involved a few valuable questions worth sharing.

Read More
Published by Alex Slotnick on Dec 19, 2016 6:02:58 PM

The Virtues of a Code Slush

People tend to look for sturdy, expansive safety nets when facing risky situations. In database management, organizations turn to the sweeping strategy of code freezes during times of hectic, peak activity. Most engineering teams have had to deploy code freezes during such times, with the noble intention of protecting the integrity of a system, using the logic that if nothing can touch it, nothing can break it.

Unfortunately, a code freeze is a flawed strategy.

Read More
Published by Alex Slotnick on Dec 9, 2016 6:23:11 PM

Everyday Profiling and Calculating Query CPU

When you find spikes in your database's CPU consumption, locating the cause behind those peaks can be a daunting task. Database server software doesn’t make it easy to precisely analyze what's taxing CPU and IO, and, due to the system's complexity, locating the source of the burden on system resources isn't just a matter of pulling metrics from the database and connecting the dots. These figures must be calculated.  

Read More
Published by Alex Slotnick on Dec 2, 2016 3:57:58 PM

VividCortex Newsletter: November 2016

The November 2016 edition of the VividCortex newsletter includes our Database Performance Monitoring Buyer's Guide; PayLease's video case study about their experience with VividCortex; a recording of our "Five Things to Monitor After Black Friday and Cyber Monday" webinar; a free Square Root Staffing Law workbook; and more.

Read More
Published by Alex Slotnick on Dec 1, 2016 3:12:01 PM

Webinar Recap: Five Things to Monitor After Black Friday and Cyber Monday

If you missed this week's broadcast of "Five Things to Monitor After Black Friday and Cyber Monday," a recording is now available for on-demand access. Why's it important to carry your monitoring principles beyond Black Friday weekend? Even though the holiday shopping spree has slowed down, organizations can continue to monitor their databases wisely. Don't be fooled: database monitoring is vital year-round, and retail organizations should be prepared to deal with critical activity--even during periods unrelated to the typical holiday surges.

Read More
Published by Alex Slotnick on Nov 28, 2016 5:22:23 PM

AWS re:Invent 2016 Survival Guide

AWS re:Invent 2016 kicked off this morning, and attendees of the conference are in for a week overflowing with breakout sessions, workshops, entertainment, and cloud computing innovation, all at the heart of the Las Vegas Strip. We know from firsthand experience that conferences like AWS can be exhausting even as they're exhilarating, and attendees will absolutely benefit from some smart planning and preparation, to ensure that they get the most for their time, without feeling they missed anything major. So what does it take to make sure AWS delivers as much as possible? We present this brief—but useful!—survival guide to help you navigate this year's schedule and events.

Read More
Published by Alex Slotnick on Nov 23, 2016 5:48:41 PM

Monitoring for Black Friday: Capacity Planning Techniques

In our latest webinar, Baron Schwartz talked about how to best prepare for potential traffic spikes during Black Friday and Cyber Monday. Although some organizations may not experience a seasonal peak during the holidays, there might be other times when traffic surges can be expected--we thought we’d use the occasion of Black Friday to share some techniques to help you prepare for times when your systems may be stressed.

Read More
Published by Baron Schwartz on Nov 18, 2016 2:16:00 PM

Why Percentiles Don’t Work the Way you Think

Customers ask us for p99 (99th percentile) of metrics pretty frequently.

We plan to add such a feature to VividCortex (more on that later). But a lot of the time, when customers make this request, they actually have something very specific, and different, in mind. They’re not asking for the 99th percentile of a metric, they’re asking for a metric of 99th percentile. This is very common in systems like Graphite, and it doesn’t achieve what people sometimes think it does. This blog post explains how percentiles might trick you, the degree of the mistake or problem (it depends), and what you can do if percentile metrics aren't right for you.

Read More
Published by Preetam Jinka on Nov 9, 2016 10:15:00 AM

What is ACID and How Should I Think About It As a Developer?

For the average reader, the phrase “expert in ACID” most likely evokes a picture of either an eminent chemist or a jolly Deadhead. However, in the world of computer science, an understanding of "ACID" means something a bit different (though not mutually exclusive!). When dealing with relational databases, the term refers to a set of properties that describe certain characteristics of transactions.

Read More
Published by Karen Bender on Nov 7, 2016 10:05:08 AM

Why VividCortex Observes Election Day as a Holiday

In Abraham Lincoln’s words, the United States is a “government of the people, by the people, for the people.” Not a select few, not people in just one area of the country, not only wealthy and privileged people, but all people.It's therefore our duty and responsibility to elect the officials who represent us by casting our votes on Election Day.

Most states require employers to give employees at least two or three hours off to vote. Some states require paid time off; others do not. Twenty states and the District of Columbia do not have specific laws that require that workers be given time off, either paid or unpaid, to vote. Virginia, where VividCortex is headquartered, is one of those states. Although we have an open paid time off policy and employees can use that time to vote, we have decided to close our offices on Election Day. Why?

Read More
Published by Alex Slotnick on Oct 31, 2016 4:48:45 PM

5 Questions Managers Should Ask Their DBAs Before Black Friday

Poorly performing sites can be costly. With Black Friday and Cyber Monday right around the corner, how can business leaders work with DBAs to ensure that their organizations and databases are properly prepared to deal with customers’ needs, so websites don’t malfunction? With five simple—but powerful—questions, managers can help their database teams get ready for the heaviest of shopping traffic.

Read More
Published by Alex Slotnick on Oct 28, 2016 4:18:11 PM

VividCortex Newsletter: October 2016

The October 2016 edition of the VividCortex newsletter includes information about our Role-Based Access Control and Single Sign-On features; our Universal Scalability Law Modeling Workbook; a recap of a webinar about PostgreSQL 9.6 and MySQL 8.0; information about our upcoming events; and more.

Read More
Published by Baron Schwartz on Oct 28, 2016 10:43:44 AM

Everyone's Capable of a Performance Mindset

The other day I was speaking with someone who asked me a few times how we find and hire engineers who are capable of building performance/monitoring tools. I didn't have a great answer in the moment, but after the conversation was over, I realized it's because I fundamentally believe we all are capable of understanding and solving systems performance problems. We're all brainiacs, so to speak.

Read More
Published by Alex Slotnick on Oct 27, 2016 6:26:42 PM

Webinar Recap: What's New in PostgreSQL 9.6 and MySQL 8.0?

Earlier this week, Baron Schwartz, VividCortex's CEO and author of High Performance MySQL, led a thorough webinar about the new and upcoming updates for a couple major databases: MySQL 8.0 and PostgreSQL 9.6. Attendees of the webinar had the opportunity to learn and ask questions about the most exciting, most impactful, and most overdue changes on offer in these latest iterations.

Read More
Published by Alex Slotnick on Oct 25, 2016 10:26:53 AM

"Agents Maintenance Windows" Balance User Control With Performance

It should go without saying that we want all of our customers to have access to the latest, most up-to-date components of VividCortex. With that in mind, we've designed the VividCortex app to include default settings for automatic updates. But what if users want to take control over their systems' update schedules? VividCortex has offered solutions in the past, but with a new feature--Agents Maintenance Windows--we're giving users even more precision in maintaining their systems.

Read More
Published by Alex Slotnick on Oct 17, 2016 2:41:18 PM

Smart RBAC and SSO Implementation Improves Security, Management, and Workflow

VividCortex's recent introduction of two important product features—Role-Based Access Control (RBAC) and Single Sign-On (SSO)—gives us the opportunity to initiate a conversation about topics that are relatable to anybody at a modern-day company or organization: security, workflow, and effective management. How can a powerful monitoring solution improve the way an organization operates while also making the way it handles data safer?

Read More
Published by Kelsey Uebelhor on Oct 13, 2016 9:45:57 AM

New User Security and Compliance Enhancements with RBAC and SSO

We’re thrilled to announce a brand new set of capabilities available in VividCortex that address an ongoing challenge for users, and engineering and application team managers alike – provisioning and managing users.

While users have come to expect powerful database monitoring insights and visibility from VividCortex, we’re now matching those high standards in how we meet the needs of enterprises in compliance, security, and user management.

Read More
Published by Alex Slotnick on Sep 29, 2016 6:48:13 PM

Upcoming Database Updates: A Very Brief Pronunciation Guide

As many readers already know, it's update season for a couple key database systems. This morning the PostgreSQL Global Development Group released PostgreSQL 9.6 and published an announcement covering need-to-know details. Similarly, MySQL 8.0 is right around the corner, and as the full numeric uptick to 8.0 indicates, it's bound to contain some significant changes. This is major news for many users of these platforms. In recognition of these shifts in the database landscape, we'll be hosting a webinar on October 25, where Baron Schwartz will look at the changes these updates contain, discussing features such as replication capabilities; more extensibility; improved performance; broader SQL implementation; and more. 

In the meantime, we wanted to take a moment to help prep users in a different way: this blog post is a quick, pocket pronunciation and etymology guide that we hope will assist readers in uttering the vowel-phobic names of these systems with confidence.

Read More
Published by Preetam Jinka on Sep 28, 2016 3:43:12 PM

Optimizing VividCortex’s Samples View: Part 2

A few weeks ago, I wrote a blog post explaining how sketch sampling methods can be employed to achieve fast 30-day data visibility for monitoring users. The problem we faced was that with that standard of retention, we’ve frequently seen systems that involve nearly a million query samples in a 30-day window, meaning that special solutions are needed in order to avoid overloading users’ browsers.

The solution we’ve found lies in a hash ordering that's proven to be both surprisingly simple and efficient. In this Part 2 post, I’ll look at why it works so well.

Read More
Published by Alex Slotnick on Sep 28, 2016 2:48:05 AM

Webinar Recap: Monitoring with Percentiles

Earlier today, Baron Schwartz hosted a live webinar that took a deep look at percentiles and their role in performance monitoring. This is a tricky, engaging subject: as Baron wrote in a blog post last year, "Customers ask us for p99 (99th percentile) of metrics pretty frequently" -- but percentiles often don't work the way many people think.

You can access a recording of today's webinar here, where Baron returned to that subject and illuminated many of the finer points of percentiles, this time examining why people ought to be careful around percentiles and how people can best use them to their advantage.

Read More
Published by Preetam Jinka on Sep 23, 2016 3:32:59 PM

How Color Schemes and Query Latency Bands Help With Database Monitoring

In this post, we’ll explore some powerful visualizations that are vital to monitoring, and we’ll take a look at a few of the attributes that make them so effective—specifically, color schemes. Colors are a central element for many effective visual tools, and whether colors are used to communicate information more clearly; to emphasize certain nuances of data; or to achieve an aesthetic or emotional response, they are a major part of a visual’s impact.  To keep things simple, this blog post will focus our examination on metrics that show us how work is distributed.

Read More
Published by Alex Slotnick on Sep 19, 2016 5:06:13 PM

Upcoming Webinar: 8 Ways to Win with Percentiles

Last year, VividCortex’s CEO Baron Schwartz published a compelling exploration about one of the concepts in database monitoring that we see most often discussed—and most often misunderstood: percentiles. “Why Percentiles Don’t Work the Way You Think” instantly became one of our most popular blog posts ever, as it helped introduce readers to the nuances surrounding some of the fundamental, widespread, and hazardous misconceptions that people apply to percentiles and percentiles' role in database monitoring practices.

Read More
Published by Baron Schwartz on Sep 16, 2016 1:19:26 PM

Surprising Behaviors of PostgreSQL Statistics Views

PostgreSQL provides a variety of statistics views that you can use to see what’s happening in the database. However, as with any complex system operating under a demanding workload, using these views may cause unwanted performance side effects, if you’re not careful. At the same time, the views have special behaviors that might surprise you if you’re not aware of them. If you’re planning to examine the PostgreSQL statistics views for any purpose, here’s what you should watch for.

Read More
Published by Alex Slotnick on Sep 12, 2016 5:11:24 PM

Queueing Theory in the News: NYT Asks "How do You Pick the Fastest Line?"

Just in time for the release of the newest edition of our popular, free ebook -- "The Essential Guide to Queueing Theory" -- the New York Times has published an article that delves straight into the heart of one of the most common daily-life instances of queueing-theory-in-action: titled "How to Pick the Fastest Line at the Supermarket," it investigates the grocery store line, that most common and well-known of quotidian queues.

Read More
Published by Alex Slotnick on Sep 7, 2016 5:40:04 PM

iPhone 7, Headphone Jacks, and How Apple Hijacks Time-Series Data Trends

All ye faithful: the iPhone 7 is nigh, just as all tech blogs foretold. This afternoon, Apple kicked off September in the same way it has for the past several years, by making a product announcement and adding new technology to their market-warping and trend-setting lineup. As early Fall has become synonymous with Apple’s unique brand of press conference, these announcements have attained a reputation as faux-holiday, demanding live coverage across the internet, from Wired to The New York Times to The Guardian to The Times of India.

Read More
Published by Alex Slotnick on Sep 2, 2016 11:54:36 AM

Webinar Recap: VividCortex and Datadog Tackle "Work-Centric" Monitoring

Earlier this week, VividCortex's Preetam Jinka co-hosted a webinar alongside Datadog's Matt Williams. Together, they tackled one of the most important questions for effective monitoring practices: how to define what you should be looking at in your systems to get the most value for your efforts. If you missed the original broadcast of "5 Tips on Determining the Most Impactful Metrics in Your App," a free recording of the 40 minute webinar is now available, so you too can reap the benefits of Preetam and Matt's monitoring insights. 

Read More
Published by Baron Schwartz on Sep 1, 2016 11:14:28 AM

Auditing Query Activity with VividCortex

Did you know you can use VividCortex to pull needles out of haystacks with super low effort and overhead? It’s true: next time you’re wondering how to catch that oddball query in action, without enabling a month’s worth of full query logging, VividCortex might have a solution for you.

Read More
Published by Preetam Jinka on Aug 25, 2016 4:48:32 PM

Optimizing VividCortex’s Samples View: Part 1

Deterministically Subsampling Queries: A Million Samples?!

VividCortex lets users drill-down into their data up to a single-second granularity, and, using specialized sketch sampling methods (check out our free ebook about Sampling a Stream of Events With a Probabilistic Sketch for a look at one way we employ sampling to handle big data sets) we’re able to offer customers up to 30 days of data retention. As you can imagine, 30 days of samples represents a lot of data. Exactly how much?

Read More
Published by Alex Slotnick on Aug 15, 2016 11:39:26 AM

Doing Work with the Law that Defines Critical APM Metrics

Later this month, on August 30th, VividCortex’s Preetam Jinka will join Datadog’s Matt Williams for a webinar to discuss 5 Tips on Determining the Most Impactful Metrics in Your App,” along with the solutions each company uses to track those metrics.

In anticipation of that conversation, this blog post goes a bit behind the scenes and introduces the powerful theorem Little’s Law, based in the theory of probability, and the ways it provides the ability to define a set of metrics that we value as highly critical for gaining database insights.

Read More
Published by Alex Slotnick on Aug 8, 2016 3:41:44 PM

Fault Detection Released for MongoDB

As we noted last week on our product updates page, VividCortex has recently added fault detection support for MongoDB. Adaptive fault detection is one of VividCortex’s unique and most powerful features: it’s a practical and useful application of the concept of “work-centric” monitoring. Rather than focus on global resource metrics, which can often be relative and context-dependent, work-centric measurements give a clear and proactive perspective on what your system is really up to. We’re excited to be introducing this key resource to MongoDB users.

Read More
Published by Alex Slotnick on Aug 4, 2016 2:23:23 PM

5 Monitoring Myths That Deserve to Be Busted

There’s no doubt that effective application performance management can be a daunting goal. As businesses have demanded increasingly complex tasks from their technology, the solutions required to keep their systems in top shape need to be ever more insightful and precise. While things like efficiency and uptime are the bottomline indicators of an app or database’s performance, there’s a variety of potential methods that can be used to achieve those ideals.

As in any other industry, false information proliferates where people are eager for quick, simple solutions. Naturally, we all want the fastest, most direct ways to fortify our APM. But where ambitious ideas can be powerful and positive, they can also be too good to be true or send the wrong message. Whether these involve — for example — overestimation of percentiles’ application or misunderstanding root-cause analysis, such expectations are pernicious in how they lead people to look in the wrong places or misuse the information they have at hand.

We’ve encountered a few of these misconceptions that stand out from the rest — here are 5 monitoring myths that deserve to be busted.

Read More
Published by Alex Slotnick on Jul 28, 2016 4:02:59 PM

SQL Injection Detection and Alerting are Vital For Secure Data

The recent Mossack Fonesca “Panama Papers” hack is the latest security breach to drive home how much an impact an SQL injection can have on modern-day organizations. Though that hack ultimately revealed massive professional fraud by companies and governments around the world, it involved the exposure of 11.5 million confidential documents. The 2.6 terabytes of data stolen were a powerful reminder that the history of web-based business has been riddled with instances of SQL injections. They remain common and potentially devastating to organizations.

Read More
Published by Alex Slotnick on Jul 9, 2016 12:41:44 PM

VividCortex Doesn’t Do Root Cause Analysis... and That's a Good Thing

Root cause mentality is the idea that when an issue appears within a system, it’s always possible to find a single, underlying reason why, if you just dig deep enough.

Much of the time, this approach to problem-solving makes sense, it feels intuitive, and it leads to efficient solutions. However, it’s a rationale that doesn’t hold up universally — especially when facing problems of higher complexity. When applied to database monitoring, for instance, this logic might lead people to assume that an “ideal” monitoring product should come equipped with powerful root cause analysis: the ability to seek out, identify, and point to the root cause of an issue within the database system. From there, in theory, crafting an effective solution should be easy. Unfortunately, this approach seriously overlooks the actual complexity of modern-day systems. And that oversight can be costly.

Read More
Published by Alex Slotnick on Jul 7, 2016 4:44:43 PM

Why Adaptive Fault Detection is Powerful and Unique

Adaptive Fault Detection is a patented, algorithm-based technology and one of the important central components of the VividCortex app . Unlike other monitoring methodologies  such as anomaly detection or threshold alerting — adaptive fault detection is designed to detect events that are, by definition, detrimental to a system. It looks for issues that actually prevent work from completeing — not just anomalies or outliers. With this quick blog post, we want to help readers understand the definition and value of fault detection. To do so, it helps to delve into several key concepts:

Read More
Published by Alex Slotnick on Jul 5, 2016 10:04:32 PM

Databases in Culture: Brexit Edition

The UK voted in favor of Brexit on June 23rd, and, in the weeks since, the world has watched with bated breath to see what impacts and developments we should all expect. What does Brexit mean, people want to know, for sectors like finance, immigration, tourism, and beyond? How wide-sweeping is the decision to break away from the EU, a 40-year-old bloc?

Here at VividCortex we also can't help but wonder, How can we understand Brexit in terms of the databases surrounding it? As we've looked at in previous blog posts, databases play central, constant roles in our modern day culture, in everything from hurricane tracking to modern art museums, from election finances to music streaming services. If there's something in the news about how our culture is changing, there's a good chance that some sort of data system is also close at hand. For global news as massive as Brexit, we're curious about the ways in which relevant databases are changing and affecting the people connected to them.

Read More
Published by Ewen Fortune on Jun 28, 2016 3:39:30 PM

5 Database Insights Easy to See with VividCortex SaaS Monitoring

There are manifold ways to collect, visualize, and analyze data… but not all methods are equally useful. VividCortex, however, is singular as a database-centric SaaS monitoring platform, and it's designed to provide you with powerful insights into your system that are both inherently actionable and unique. Within minutes of first booting up VividCortex, users frequently discover new aspects of their system. They understand it in brand new ways, just by viewing our app’s basic dashboards and metrics.

Read More
Published by Alex Slotnick on Jun 17, 2016 5:31:11 PM

Using VividCortex to Help Optimize MySQL Configuration

Many users new to MySQL will find themselves overwhelmed when tasked with configuring their system for best performance. It doesn't need to be intimidating, however. MySQL offers nearly 400 settings to consider, but a user should understand that most of the system's benefits can be achieved just by getting a few fundamental configurations right. The vast majority of optimizing configuration comes from selecting those very important options. Knowing where to look is the first step, and utilizing tools such as a monitoring product can be a huge boon.

Read More
Published by Alex Slotnick on Jun 14, 2016 10:58:30 AM

How VividCortex Achieves Gapless Aggregator Restarts

As many users already know, we run a set of agents in each box of each customer's environment. All but two of these agents are set up to continuously fetch statistics from the connected OS and the different database instances that the agents are able to find.

Read More
Published by Alex Slotnick on Jun 9, 2016 12:11:13 PM

VividCortex's SQL Injection Detection

For data-driven applications, security is of absolute importance. Virtually all modern tech-driven organizations must treat the protection of sensitive information as an imperative. At VividCortex, we know that our customers value the premium we put on data-security, along with the specific defenses we have in place that keep their data out of the wrong hands.

Read More
Published by Owen Zanzal on Jun 6, 2016 7:14:50 PM

Goodbye, Incidents. Hello, Alerts.

All VividCortex users, please repeat: “Goodbye, incidents. Hello, alerts.”

We’re changing the way that VividCortex tells you when you need to pay attention to a specific occurrence in your system. We recently phased out VividCortex’s “incidents” services, and the “incidents” section in our app’s UI is gone. This isn’t just a cosmetic change. We’re migrating the incident configurations into a whole new concept: alerts.

Read More
Published by Alex Slotnick on May 25, 2016 3:23:46 PM

Why Use Slow Query Logs When There’s VividCortex?

At VividCortex, we believe that database monitoring can be much, much better than what many people currently settle for. In fact, we know that monitoring needs to be better -- more reliable and more efficient than what many engineering and operations teams currently use. Databases are getting larger and more complex all the time, and older monitoring techniques can’t properly optimize modern systems or keep up with issues when they arise. For instance, we consider slow query logs a prime example of a monitoring tool that’s become virtually obsolete in a world where you can find an alternative such as VividCortex.

Read More
Published by Miguel Trias on May 20, 2016 2:38:45 PM

Common UI Performance Problems in Angular Apps

In this post, we analyze some performance issues that we recently fixed in our web app and that are common in any AngularJS app. We also discuss the tools we used to find and diagnose the causes and the strategies we used to fix these problems.

Read More
Published by Gustavo Kristic on May 6, 2016 5:41:00 PM

Addressing a Problematic MySQL Query and its Effects on Amazon Aurora

Since we announced our support for Amazon Web Service's Aurora last year, we've had great experiences working with customers who use Aurora as their database system of choice. And not only is Amazon's RDS excellent, so is their team and support.

Recently, we encountered a MySQL bug that was, in turn, having effects on Amazon’s RDS. The MySQL issue involved a problematic query that caused Aurora instances to crash under load; upon discovering this issue, we had the opportunity to work with AWS to identify the error in MySQL and fix it, so that it would avoid an impact on AWS’s systems. It turned out that the query itself posed a potential problem for all monitoring services -- not just VividCortex; fortunately, however, once AWS addresses the bug, the problem will be fixed for all monitoring products that might come across it, in regards to Aurora. Here's some background on the issue and how we addressed it.  

Read More
Published by Preetam Jinka on May 4, 2016 1:44:52 PM

How VividCortex Uses the New Generated Virtual Columns Feature in MySQL

In an industry as fast-growing and interconnected as database technology’s, it’s exciting to track how innovations in one platform can ignite beneficial, rippling effects on other, surrounding systems. At VividCortex we frequently find ourselves faced with opportunities to improve our monitoring solutions based on how database technologies (such as MySQL, Redis, PostgreSQL, etc) develop and integrate new upgrades.

Read More
Published by Alex Slotnick on Apr 27, 2016 12:01:34 PM

Charitable Donations to Diversity-in-Tech

As easy as it is to go deep and get lost in the technical, nitty-gritty side of database monitoring, at VividCortex we prioritize always keeping at least one eye open to the world around us. This means we care about the health and happiness of our employees and friends day-to-day, in our office and at home; we care about striking a balance in how technology fits into our lives; and we care about equality and awareness within our industry at large.

Read More
Published by Alex Slotnick on Apr 22, 2016 1:38:51 PM

Best Monitoring Practices, Greatest Hits

Over the last couple years on the VividCortex blog, we've discussed a huge range of database monitoring concepts. We've examined ideas we're excited about, shared monitoring practices we recommend, and explored some of the principles that shape our own projects and goals. Of these many posts, however, there are a few in particular worth emphasizing and discussing again and again. Generally, these posts explore the math and statistics behind key monitoring practices and they champion why smart database monitoring is so important; they're often our most read articles, the most indicative of what we value, the influences that make VividCortex a unique product.

Read More
Published by Alex Slotnick on Apr 15, 2016 3:54:00 PM

VividCortex's Open-Source Documentation

We were inspired by this recent post on CircleCI's blog, highlighting the many virtues of well-written and accessible documentation. Just as significantly, they point out the deeper value of making that documentation open-source. Documentation is where an engineering team can really elucidate its ideas and the execution that makes those ideas function; as CircleCI explains, it can be an invaluable resource for newcomers and veteran users alike. That sort of community inclusion is something we strongly agree with at VividCortex.

Read More
Published by Baron Schwartz on Apr 12, 2016 9:20:26 AM

Post-Mortem on VividCortex Outage April 8, 2016

On April 8th, we had a serious outage and we lost some performance metrics from your (our customers’) systems. I know how important this is to you, and above all I’m sorry. All of us are, and we’ve worked long and hard to fix the causes as quickly as we prudently could.

Read More
Published by Alex Slotnick on Apr 6, 2016 5:50:33 PM

VividCortex Brings Developers and DBAs Together

If you want to maximize a team's productivity, special attention ought to be paid to helping DBAs and developers work together as smoothly as possible. Unfortunately, this is a relationship that's overlookedor taken for grantedfar too often. 

One of the major values that VividCortex provides an organization is the way it equips developers with new insights into their data systems, giving them a better understanding of how to work with DBAs. It affords those developers more confidence and responsibility in working within a database, thereby granting them and DBAs more freedom and independence in addressing issues quickly and effectively. And that means more good vibes all around. 

Read More
Published by Baron Schwartz on Mar 25, 2016 1:55:31 PM

Why an App-Centric View Isn't Enough

Sylvia Botros, Sendgrid’s lead DBA, recently told us about one of the ways her teams have found unique value in VividCortex. “Engineering managers and individual developers have pinged me and said, ‘Hey, I saw this in VividCortex -- what do you think?’ Some of our developers are not fully familiar with the generals of MySQL,” she explained, “but VividCortex is giving them a view into how their app is dealing with databases. And at the same time it’s teaching them DB lingo, which is good.”  

This feedback from Sylvia opens the door to one of the questions we hear a lot about VividCortex: “How is your solution different from my existing APM (Application Performance Management) tool?”It’s a great question with many technical answers, but ultimately the only answers that matter are the ones focused on results.

Read More
Published by Alex Slotnick on Mar 21, 2016 4:15:04 PM

Industry Leaders Value Database Monitoring Highly

There's a clear correlation between a company's success and how much value it places on monitoring. If you look at industry leaders across the web, you see evidence that monitoring should not be an afterthought, and it should not be treated as a low-priority concern. The habits and policies of some of these organizations demonstrate a pattern: they invest in monitoring. 

Read More
Published by Alex Slotnick on Mar 16, 2016 2:49:34 PM

New Ebook: Sampling a Stream of Events With a Probabilistic Sketch

Stream processing is a hot topic today. As modern Big Data processing systems have evolved, stream processing has become recognized as a first-class citizen in the toolbox. That’s because when you take away the how of Big Data and look at the underlying goals and end results, deriving real-time insights from huge, high-velocity, high-variety streams of data is a fundamental, core use case.

Read More
Published by Alex Slotnick on Mar 14, 2016 4:44:10 PM

Do Seconds Really Matter? Twitter's Command Center Says, "Yes"

Last week, Twitter's blog published an intriguing post about the TCC -- Twitter's Command Center -- and some of the fundamental principles driving it. The post is titled When Seconds Really Do Matter, clearly communicating an idea about visibility and resolution that we very much agree with at VividCortex. "Behind unmarked doors in office buildings around the world," the post begins, adding a dash of the clandestine to the art of monitoring, the TCC maintains unwavering standards in order to meet the strains and demands that global Twitter activity puts on their systems. Not a simple task.

Read More
Published by Alex Slotnick on Mar 2, 2016 2:06:45 PM

Sometimes a Variety of Databases is THE Database You Need

We were just leafing through the 2015 edition of The DZone Guide to Database and Persistence Management, and we noticed some interesting stats in the guide's included survey, about which we'd like to share some observations. The survey is one of the ebook's central features, and it includes feedback from over 800 IT Professionals, with 63% of those respondents coming from companies with over 100 employees and 69% with over 10 years of experience -- they represent a significant and important cross-section of our industry.

Read More
Published by Alex Slotnick on Feb 27, 2016 3:55:36 PM

Upcoming Upgrades to VividCortex's Navigational Interface

Most of the time, we post about things like Queueing Theory, SketchRanking, or monitoring best practices principles.

But today, we'd like to share something much more visible: a suite of navigational updates that we'll be introducing to the VividCortex app in the coming weeks, as we complete testing and optimization. Our interface will retain the general elements of the old design, but the changes we're showcasing here will make users' experiences with VividCortex more agile, streamlined, and intuitive.

Read More
Published by John Potocny on Feb 23, 2016 11:28:18 AM

Drinking from the Firehose: How VividCortex Compresses its Metrics

Sometimes, when working with a top-of-the-line monitoring tool, having access to the best possible visibility, granularity, and reliability is simply not enough. Sure, those features represent the ultimate goals in monitoring, but, in this case, the ends do not necessarily justify the means. What about costs? As we provide a tool that achieves ambitious bottom line functionality, we also need to pay close attention to any burden that VividCortex might impose on users’ systems. We need to mitigate and minimize those costs. And we need to make sure that VividCortex is as efficient as it is powerful... especially when the onrush of data gets intense.

Read More
Published by Alex Slotnick on Feb 8, 2016 1:13:47 PM

LCFC Site Crash: Not Using Database Monitoring is Like Not Using a Goalie

Yesterday was Superbowl Sunday, and for those of us in the USA, where the NFL championship is like a national holiday, it can be easy to forget that a different kind of football is actually the most popular sport in the rest of the world. On Saturday, soccer fans had their own scintillating match to enjoy, as the English Premier League, soccer's biggest arena, saw powerful Manchester City lose a key game to the league's serious underdogs, Leicester City F.C. Now, Leicester has become a contender to win the entire league -- in the aftermath of their upset over Man City, there's been celebration, heckling, disbelief, and soccer punditry aplenty

Oh, and in the middle of Leciester's moment of triumph, their website crashed.

Read More
Published by Alex Slotnick on Feb 3, 2016 4:59:08 PM

When Is It Too Late to Monitor?

This post is part of an ongoing series on the best practices for effective and insightful database monitoring. Much of what's covered in these posts is unintuitive, yet vital to understand. Previous posts have covered Why Percentiles Don't Work the Way You Think; how to tell If a Query Is Bad; an explanation of why, when looking at charts, you should understand that A Trendline is a Model; and Why You Should Almost Never Alert on Thresholds.

***

Imagine you're an up-and-coming retail site with ambitions to someday become the world's next Etsy or Amazon. So far you've had promising early success, but you're still growing; your userbase is relatively small, but admirably devoted. After many months' long hours of meticulous coding and visionary marketing meetings, you're comfortable with the core of your enterprise and confident in both its concept and execution. In your mind, it's only a matter of time before your site blows up -- you know it offers the best retail experience available, with a cleaner, more pleasing interface than competing retailers, plus better selection, the best site performance for your current level of traffic, and stellar customer service.

Read More
Published by Alex Slotnick on Feb 1, 2016 3:01:57 PM

VividCortex Now Available for Windows

We're excited to announce that VividCortex support for Windows is now available! With VividCortex, Windows users can benefit from the first unified suite of performance monitoring tools specifically designed for today's large-scale, polyglot persistence tier. 

Read More
Published by John Potocny on Jan 20, 2016 2:46:40 PM

Solving a Go Communications Issue with A VividCortex Agent

Back in September of last year, a customer brought to our attention an unusual behavior: they were seeing inexplicable alerts, reporting that a host was failing to send data for more than fifteen minutes. Yet when the customer checked the host a short while after, they saw that no data was actually missing. Strange, we thought, so we began to investigate.

Read More
Published by Alex Slotnick on Jan 19, 2016 1:19:36 PM

Queueing Theory: London Underground Escalator Policy Challenges Social Norms

Back in October, we published our ebook Everything You Need to Know About Queueing Theory, and as part of its launch, we blogged about one way (of the many) that queueing theory has recently shown up in the real world and the news. In that post, we shared a report showing that the London Underground behaves in ways that at first seem unintuitive, but are comprehendible if you have a handle on how queueing works. Specifically, studies suggested that when the Underground’s individual trains travelled at higher velocities, travel time for passengers actually got worse. Why? The higher individual speeds were not optimal for the particular parameters of the Underground system, and they caused bottlenecks to form at key hubs, hindering the circulation of the entire system. 

Read More
Published by Alex Slotnick on Jan 14, 2016 12:40:29 PM

We'll Create Custom Dashboards for You

One of VividCortex’s core design goals is to provide the most direct and effortless way for users to maximize their insight into their systems. And, in very practical terms, at the core of achieving that goal is VividCortex’s suite of dashboards.

Read More
Published by Baron Schwartz on Dec 21, 2015 1:47:42 PM

The Factors That Impact Availability, Visualized

We all want our systems to have high availability, but sometimes the exact meaning of “high availability” is not very clearly defined. However, availability -- like scalability, performance, and so on -- can be expressed as a mathematical function; it can be viewed in quantifiable and digestible terms. In this post, I’ll explain which parameters truly influence availability: an extremely useful concept to understand, as it enables you to focus your efforts in the right places and to actually achieve higher availability instead of just spinning wheels.

Read More
Published by Owen Zanzal on Dec 16, 2015 4:14:02 PM

8 Reasons Why ChatOps Isn't Working At Your Company

About a year and half ago I published a blog post about ChatOps at VividCortex (if you’re unfamiliar with ChatOps, that original post still serves as a good primer). At the time, we were smitten with the new technology -- I wrote, “We are huge believers of ChatOps at VividCortex… GitHub’s implementation of ChatOps is amazing and has inspired me to spearhead ChatOpts within our office… ChatOps has really revolutionized how we operate.” In the months since, I've learned a lot about implementing ChatOps so that it functions as more than just an office toy. 

Read More
Published by Alex Slotnick on Dec 11, 2015 3:37:56 PM

Webinar Recap: Best Practices for High Performing MySQL with INetU

On December 9th, Baron Schwartz (VividCortex founder and CEO), Michele Corvino (INetU Product Manager), and Jim DeHart (INetU Systems Engineer) joined up to discuss best practices for keeping MySQL databases at peak performance. INetU – the best customer-rated cloud hosting company in the industry – knows the importance of keeping databases running without hitch. Complexity within systems increases as applications grow, which means that as complications arise, there are more risks for end users. Leveraging a database performance tool like VividCortex allows users to take care of such issues as quickly as possible, while optimizing efficiency at the same time.

Read More
Published by Alex Slotnick on Dec 9, 2015 4:55:47 PM

PGConf SV 2015 - Baron Schwartz on VividCortex's PostgreSQL Sniffer

PGConf Silicon Valley was excellent this year: good organization, great speakers, and top-notch vibes in general. We were excited to see everybody visiting (and crowding!) our booth and the overall very positive community presence. Highlights included a presentation about Postgres' use at TripAdvisor, a look at Amazon RDS for Postgres, and consideration of system scalability and the future of relational database management systems.

Read More
Published by Alex Slotnick on Dec 3, 2015 2:38:40 PM

Percona and VividCortex Collaborate to Provide Leading Database Solutions

We're thrilled to announce VividCortex's new partnership with Percona -- the only company that delivers enterprise-class solutions for both MySQL and MongoDB across traditional and cloud-based platforms. As part of the partnership, Percona will resell VividCortex as a value-added monitoring solution, and Percona's own support technicians and managed service administrators will have VividCortex at their fingertips, to assist in serving and optimizing their customers’ applications.  

Read More
Published by Baron Schwartz on Dec 3, 2015 2:20:45 PM

Is That Query Bad?

This post is part of an ongoing series on the best practices for effective and insightful database monitoring. Much of what's covered in these posts is unintuitive, yet vital to understand. Previous posts have covered Why Percentiles Don't Work the Way You Think; how to avoid getting to a point When It's Too Late to Monitor; an explanation of why, when looking at charts, you should understand that A Trendline is a Model; and Why You Should Almost Never Alert on Thresholds.

***

One of the most frequent reactions people have after installing VividCortex is "I didn't know my systems were doing that!" Another way to say this is, new VividCortex users almost immediately learn something new about their systems.

Read More
Published by Alex Slotnick on Dec 2, 2015 4:10:19 PM

Could Target Have Avoided its Cyber Monday Systems Shake-up?

Black Friday and Cyber Monday have now come and gone, and in the midst of the holiday's heavy shopping traffic, we hope that everything went smoothly for online retailers and their databases. As we wrote back in October, the days after Thanksgiving are famous for being the most major shopping days of the year, and they allow stores across the country to make enviable profits... if their systems are ready for the huge spikes in traffic. The best way for businesses to prep is to outfit their databases with powerful monitoring tools, so they can address any issues that arise, smoothly and swiftly. 

Read More
Published by Alex Slotnick on Nov 30, 2015 2:39:41 PM

The Power and Ease of Adaptive Fault Detection

Adaptive Fault detection is a prime example of how efficiently VividCortex can help you understand and optimize your system. We define a “fault” as a special kind of system stall: a period during which applications are asking the server to perform a great deal of work, but that work is getting bottlenecked and therefore not completing.

Read More
Published by Baron Schwartz on Nov 30, 2015 12:18:56 PM

A Trendline is a Model

This post is part of an ongoing series on the best practices for effective and insightful database monitoring. Much of what's covered in these posts is unintuitive, yet vital to understand. Previous posts have covered Why Percentiles Don't Work the Way You Think; how to avoid getting to a point When It's Too Late to Monitorhow to tell If a Query Is Bad; and Why You Should Almost Never Alert on Thresholds.

***

Excel makes it easy to add a “trendline” to a chart, but does the trendline actually reflect the processes that produced the data? Usually not. Usually a trendline is just chartjunk and you shouldn’t use it.

Read More
Published by Miguel Trias on Nov 30, 2015 12:17:05 PM

Introducing the Profiler Tool

For a long time we’ve been dissatisfied with the fact that we had two separate tools to do the same thing: Top Processes and Top Queries rank processes and queries respectively by a metric of your choice. For example you may rank queries by frequency, or rank databases by total execution time. There’s a lot of shared functionality and at the same time there’s so much more we could do with the metrics we’re capturing from your systems.

Read More
Published by Baron Schwartz on Nov 30, 2015 12:06:36 PM

A Gratitude List

A gratitude list is one of my favorite ways to influence my mood. Here’s a gratitude list on Thanksgiving Day 2015 (United States) to express some of the reasons we are so fortunate at VividCortex to have great customers, employees, investors, advisors, friends, vendors, community, and more.

Read More
Published by Alex Slotnick on Nov 24, 2015 12:17:40 PM

Entrepreneur: 4 Secrets to Giving a Speech Worthy of Standing Ovations

VividCortex’s powerful monitoring specializes in organizing, indexing, and optimizing your data and database systems – but our founder and CEO Baron Schwartz also has some sterling ideas on how information can most effectively be communicated outside those systems. In the fast, idea-driven world of tech and business, a leader’s ability to deliever a speech is vital.

Read More
Published by Alex Slotnick on Nov 20, 2015 5:01:57 PM

The Value of Log Structured Merge Trees

Indexes are usually built by way of a data structure; typically, that structure takes the form of a “tree.” Most commonly, the structure of choice is a B-Tree, which is a hierarchical organization defined by the arrangement and interactions of its nodes.

Read More
Published by Baron Schwartz on Nov 11, 2015 10:48:43 AM

Introducing Query Anomaly Detection

Anomaly detection sure is a hot topic. We’ve written about it ourselves a number of times, and Preetam Jinka and I just coauthored a book for O’Reilly called Anomaly Detection For Monitoring. One of the challenges, as we’ve discussed so often, is that catch-all, generic anomaly detection is hard to do.

Read More
Published by Baron Schwartz on Nov 6, 2015 12:37:00 PM

Nobody Loves Graphite Anymore

Many of our customers use Graphite, and I don’t think anyone would argue with me when I say it’s probably the most commonly used time series database in the DevOps community. Not only does it have a huge installed base, it also has a robust community of advocates and developers, including Jason Dixon, who’s writing a book on monitoring with Graphite. (We featured Jason in our Brainiac Corner a while ago.)

Read More
Published by Alex Slotnick on Nov 4, 2015 11:44:38 AM

VividCortex on Fast Company: Four Signs You're Micromanaging Your Employees

We’re thrilled to point out that VividCortex CEO Baron Schwartz has had a second article published through Fast Company this week. This one, titled “Four Signs You’re Micromanaging Your Employees,” elaborates upon a few telltale behaviors that indicate when a manager is getting overinvolved with employees’ workflow. Such micromanagement can easily do far more harm than good, but its often difficult to see when you’re doing it yourself. Baron is careful to point out, however, that when a manager understands the limits of micromanaging, it can lead to great opportunities for strong work all around.

Read More
Published by Alex Slotnick on Oct 28, 2015 3:36:35 PM

VividCortex on Fast Company: How Remote Work Policies Can Backfire

Baron Schwartz, VividCortex’s founder and CEO, has published an article on Fast Company’s website, about “How Remote Work Policies Can Backfire.” Baron offers an insightful perspective on how technology and telecommunication has changed the way people interact in work settings and how remote work might have effects that are not often discussed – in some cases, remote work might come at the cost of a company’s sense of community.

Read More
Published by Baron Schwartz on Oct 26, 2015 3:38:27 PM

Beware of Database Tuning Advisors

One of the common questions we get during sales demos is “does VividCortex give advice on my database’s configuration?” The assumption is that since our product sees lots of information about the database, operating system, and current configuration, it can “optimize” the database configuration. Or, at least, point out really obviously bad things? Surely that is not too hard to do?

Read More
Published by Alex Slotnick on Oct 22, 2015 2:11:40 PM

How to Turn Black Friday Pandemonium into Profit — Buff Up Your Databases

Black Friday means two major things to retailers in 2015: opportunity and chaos. These ideas go hand-in-hand, of course. On the biggest shopping weekend of the year, retailers find themselves faced with the opportunity to benefit from the demands of wave after wave of customers – if those waves don’t crush them first. News stations across the country delight in covering the annual craziness at retailers’ physical locations, and over the past few years, many of us have gotten used to watching highlight clips of shopping anarchy, the morning after Turkey Day, while we recover from our tryptophan comas.

Read More
Published by Baron Schwartz on Oct 15, 2015 12:56:20 PM

What If You Can't Trace End-to-End?

A lot of monitoring systems have a goal of end-to-end tracing, from “click to disk” or something similar. This is usually implemented by adding some kind of tracing information to requests. You can take a look at X-Trace or Zipkin for a couple of examples. The idea is that you can collect complete traces of the entire call tree a user request generates, even across services and through different subsystems, so a slow web page load can be blamed on an overutilized disk somewhere.

Read More
Published by Alex Slotnick on Oct 2, 2015 11:56:17 AM

Mastering the Sort Buffer and sort_buffer_size

The sort buffer, which is controlled by the sort_buffer_size setting, is probably one of the most-discussed MySQL settings. Interestingly, in many cases it has a lot less potential to help performance than to hurt it, so configuring it is often about avoiding trouble rather than gaining a lot of performance.

Read More
Published by Alex Slotnick on Sep 30, 2015 1:50:59 PM

Hurricane Season? Here's Some Hurricane Data

October is almost here, and fall officially started last week; pumpkin in all its natural and synthetic forms has initiated its perennial invasion; and the more breezy among us might assume that the potential for tropical storms went out the window along with the summer’s tropical weather. Unfortunately, that’s not the case – hurricane season officially lasts until November 30 – and, just this morning, Hurricane Joaquin turned its eye toward potential landfall on the United States’ Atlantic Coast.

Read More
Published by Alex Slotnick on Sep 28, 2015 6:09:24 PM

A Few Fundamental Rules for Enlightened Database Monitoring

It’s a pretty fair assumption that if your database is big enough and complex enough to produce metrics that warrant a monitoring system, it’s also complex enough to produce tons of data that are ultimately more distracting than relevant. It’s not unusual to look at a bevy of monitoring possibilities and feel overwhelmed, uncertain about where to center your focus. Of course, every database is different, but there are some fundamental truths you should consider when you ask yourself, “What should I monitor?” Some of these ideas might seem simple, but if you don’t keep these in mind, you’d be surprised how easy it can be to lose sight of the big picture.

Read More
Published by Alex Slotnick on Sep 25, 2015 1:09:35 PM

Selecting Queries for Optimization

Query optimization can feel like an overwhelming task. Where to start? In this blog post, we’ll discuss how to select which queries you really should optimize. This is a step that’s often skipped, but shouldn’t be! Why is selecting the right queries to optimize so important? There are several reasons.

Read More
Published by Alex Slotnick on Sep 25, 2015 11:42:17 AM

ReadWrite: Why DBaaS Will Be The Next Big Thing In Database Management

The Database-as-a-Service (DBaaS) model is already making a big impact both in the tech world and in industries beyond. What do you need to know about DBaaS? Over on ReadWrite, VividCortex’s CEO and founder Baron Schwartz has published an article that considers the powers of DBaaS, how the model fits into the current state of database technologies, and how it will continue to grow and make big waves.

Read More
Published by VividCortex on Sep 15, 2015 5:42:00 PM

Brainiac Corner with Ryn Daniels

Ryn Daniels is an operations engineer at Etsy who got her start in programming with TI-80 calculators back when Geocities was still cool, and these days has opinions on things like ‘the cloud’ and ‘devops’. She lives in Brooklyn with a perfectly reasonable number of cats, and in her spare time can often be found homebrewing, rock-climbing, or hand crafting knitted server koozies for the data center.

Read More
Published by Alex Slotnick on Sep 11, 2015 11:18:57 AM

Inefficient Internal Systems Lead to Inefficent Administrators

Everybody knows that the amount of data around us is constantly growing, and yet database administrators still often find themselves ill equipped to perform their jobs effectively. If administrators only have access to antiquated techniques and tools, an entire company’s profitability and productivity can be affected.

Read More
Published by Alex Slotnick on Sep 9, 2015 11:01:48 AM

VividCortex on Tech.Co: Can Your Business Benefit From a Cloud Database?

Should you move your database to the cloud? Over on Tech.Co, VividCortex's CEO and founder Baron Schwartz has published an article that covers some of the major benefits of such a shift. Cloud computing has changed tremendously over the past decade, and the potential for databases in the cloud is only growing. Baron's article explains what some of those changes might mean for you and your data.

Read More
Published by Baron Schwartz on Sep 1, 2015 12:07:00 PM

Integrating VividCortex with VictorOps

It's easy to integrate VividCortex with your existing workflow and team tools. Included on our growing list of integrations is VictorOps. If you're not familiar with VictorOps, it's a SaaS solution designed to help manage teams' workflows after an incident is opened: it helps you handle alert routing, message delivery, on-call rotation, scheduled maintenance, escalation, and more. (If you're building these things yourself, you're probably Doing It Wrong).

Read More
Published by Baron Schwartz on Aug 31, 2015 11:32:00 AM

Integrating VividCortex with PagerDuty

It's easy to integrate VividCortex with your existing workflow and team tools. Included on our growing list of integrations is PagerDuty. If you're not familiar with PagerDuty, it's a SaaS solution designed to help manage teams' workflows after an incident is opened: it helps you handle alert routing, message delivery, on-call rotation, scheduled maintenance, escalation, and more. (If you're building these things yourself, you're probably Doing It Wrong).

Read More
Published by George Michie on Aug 31, 2015 11:16:00 AM

3 Key Ways to Make Sure Your Best Developers Are Happy

It's not easy to build a great team. And, once you've built it, maintaining such a team can be even harder. Collaboration is often the biggest challenge, along with finding the right ways to foster it. A successful organization must be more than the sum of its parts—after all, that's why we found companies, recruit around the world, pool resources, cooperate, and, as a result, accomplish great things, far beyond what any individual could do.

Read More
Published by Alex Slotnick on Aug 28, 2015 11:10:00 AM

VividCortex's Time Selector Feature

In this blog post, we continue looking at video demos of VividCortex’s key product features. Last time, we saw Top Queries in action, and today we’ll check out the Time Selector, which controls the amount of history the application displays. VividCortex always shows data for a range of time, and the Time Selector is a powerful tool that allows the user to customize that range dynamically and view data in as particular a scope as necessary.

Read More
Published by Alex Slotnick on Aug 26, 2015 10:57:00 AM

7 Databases We'd Love to See Released to the Public

Last week we blogged about how the Museum of Modern Art released a CSV file with an enormous cache of data, giving the public access to information about 125,000+ pieces of art. By sharing that data under the Creative Commons license, the museum offered the world the power to spread that knowledge without risk. MoMA’s publication of their database was a huge boon to scholars and art lovers, plus a fantastic endorsement of open information.

Here at VividCortex, we got to thinking: What are some other databases we’d love to see released to the public? Check out our list below.

Read More
Published by Baron Schwartz on Aug 20, 2015 10:40:00 AM

Send Custom Metrics to VividCortex

Did you know that you can send custom metrics to VividCortex easily? It's not something we've focused on, but it's been possible to do since the beginning.

One of our agents is a StatsD, which means you can send metrics to it easily with any StatsD client (or none--because it's easy to send metrics to it from a shell script too). The main difference between our agent and a typical StatsD is that you can send us one-second resolution data and it won't immediately fill up disks on a Graphite server somewhere and crash it.

Read More
Published by Alex Slotnick on Aug 19, 2015 10:00:00 AM

The Fine Data of Fine Art

Here at VividCortex, it’s clear to us that proper database monitoring is “an art.” Beyond simple, raw power, navigating and understanding a database takes finesse, insight, and, often, a kind of eloquence. 

On special occasions, though, the artistry of database monitoring becomes a little bit more literal, and we get to see how the functions of big data can be applied to the actual world of Fine Art.

Read More
Published by Baron Schwartz on Aug 14, 2015 11:11:00 AM

The 8 Best Ways To Lose Your DBA

As we all know, good DBAs are a dime a dozen. They’re easy to replace and the cost of replacing them in terms of lost productivity, downtime, recruiting, training, etc is minimal. You may even suspect that your DBA(s) aren’t very good since there is occasional downtime and people complain about the systems running too slowly. Firing people is icky so we’ve identified 8 great ways to encourage your DBA to leave.

Read More
Published by Alex Slotnick on Aug 12, 2015 11:16:00 AM

Breaking Databases - Keeping your Ruby on Rails ORM under Control

Object-relational mapping is common in most modern web frameworks such as Ruby on Rails. For the developer APIs, the ORM provides simplified interaction with the database and a productivity boost. However, the layer of abstraction the ORM provides can hide how the database is being queried. If you’re not paying attention, these generated queries can have a negative effect on your database’s health and performance.

Read More
Published by Baron Schwartz on Aug 11, 2015 11:19:00 AM

Facebook's Charity Majors Says VividCortex Is Impressive

A few months ago, we featured Charity Majors, the production engineering manager for Parse at Facebook, on Brainiac Corner. We are featuring Charity and her expertise once again. This time, though, she is reviewing VividCortex: from installation to problem solving to a feature wishlist.

One of our favorite takeaways: “And VividCortex is a DB monitoring system built by database experts. They know what information you are going to need to diagnose problems, whether you know it or not. It’s like having a half a DBA on your team.” And without further ado…

Read More
Published by Alex Slotnick on Aug 10, 2015 11:22:00 AM

Election Money and Data

So far, the early drama of the 2016 presidential race has been more silly than substantial. Donald Trump has been a (successful?!) one-man show, Hillary has been playing coy, some candidates have been getting trolled (much to Reddit’s delight), and, this week, Fox News has announced its first Republican debate line-up, which somehow seems more akin to the guest list for a popular 7th grader’s slumber party than presidential grooming. But there’s been one topic of conversation that’s been deadly serious and that is sure to stay on people’s minds: campaign finance.

Read More
Published by Baron Schwartz on Aug 7, 2015 11:26:00 AM

Find Queries By Error Code With VividCortex

VividCortex now lets you search for queries that cause a specific error in your application. The error code itself will be database-specific, but for example error 1062 in MySQL is a duplicate key error, and in PostgreSQL error 23503 is a foreign key violation.

Read More
Published by Baron Schwartz on Aug 4, 2015 11:32:00 AM

Inspecting All The Metrics With VividCortex

Someone asked me the other day about some of the more obscure metrics available from a database server’s internals, and wanted access to those. As it happens, we have a feature that lets you see every metric on your systems—yes, every metric, which is typically many thousands, sometimes millions.

Read More
Published by Alex Slotnick on Aug 3, 2015 11:42:00 AM

VividCortex's Top Queries Feature

In our new video, Baron Schwartz examines Top Queries, one of VividCortex’s most popular features, and how it can help find and solve performance problems inside a given application. Baron uses Top Queries to look at VividCortex’s own production systems, which involve more than fifty servers, and demonstrates the impressive level of granularity the feature makes available for examining queries.

Read More
Published by Alex Slotnick on Jul 29, 2015 5:09:00 AM

Streaming Music Services and Databases

The average American consumer isn’t usually interested in “buying” a big database. But thanks to the recent boom in the popularity of subscription music services like Spotify, databases are competing for folks’ cash more directly than ever before. After all, what are these services other than giant banks of media, made accessible and sleek? The summer of 2015 has brought the battle for music-lovers’ wallets to fever pitch, with the launch of Apple Music and Tidal, both poised to challenge the reign of top-dog Spotify.

Read More
Published by Anna Navatsyk on Jul 28, 2015 6:50:00 AM

See VividCortex in Your Server's Time Zone

Similar to databases, the modern IT team is often distributed. As remote work increases and employee locations span the globe, it becomes difficult to keep everyone on the same page. Our latest time zone update is a step toward this goal.

Read More
Published by Alex Slotnick on Jul 21, 2015 7:01:00 AM

Webinar Recap - 5 Ways to Reduce Database Infrastructure Costs

Are your databases more costly than they need to be? If you’re not sure, this webinar will show you how to find out, quickly and easily, with KPIs that indicate overprovisioning or opportunity for driving more efficiency from your databases. If there is an opportunity to run your databases at lower cost, you’ll find out how to identify which of 5 key areas offer the biggest chances for improvement, and a specific action plan to get results fast.

Read More
Published by Alex Slotnick on Jul 21, 2015 2:44:00 AM

VividCortex's 12 Most Popular Blog Posts of All Time

Here at VividCortex, it can be exciting (and humbling) to take an occasional step back and revisit some of the great work that’s been done in our offices over the past couple of years. Our blog in particular has grown quite a lot. We’re finally at a point where we can look over the steady stream of articles we’ve published and the manifold topics covered, and we can point to a few in particular and say, “Niiiiice.” But our opinions don’t count for much – it’s you, the faithful VividCortex blog readers, whose say-so really matters. What is it that you’ve found most interesting, most helpful, and most enjoyable in our archives?

Read More
Published by Baron Schwartz on Jul 20, 2015 9:24:00 AM

What Is InnoDB History List Length?

Houston, we have a problem. Google search for “What is innodb history list length?” and you get a bunch of nonsense mixed in with correct information, and it’s hard to tell which information is right. Let’s fix this.

Read More
Published by Anna Navatsyk on Jul 16, 2015 5:25:00 AM

Big Data Visualization

The Open. The historic foundation of the major championships. Played on one of the iconic courses throughout Great Britain’s lush hills. The intensity is palpable as golf’s greats compete for the coveted Claret Jug. Every drive, chip and putt matter. Players are a birdie away from glory and a sandtrip away from irrelevance, and this year, big data is at the heart of this age old tradition.

Read More
Published by Alex Slotnick on Jul 14, 2015 10:51:00 AM

Little Pluto and Its Big Data

The world’s scientific communities are abuzz today thanks to NASA’s New Horizons probe and its mission to Pluto, which achieved the closest approach of its flyby this morning and started collecting a wealth of new information on our solar system’s tiniest and most distant planet/not-planet. What did NASA share with us earthlings first? A picture, of course. The most glamorous profile ever recorded of Pluto, the King of the Kuiper Belt, graced the front pages of our newspapers and greeted readers with what will surely become an iconic shot. NASA knows that for the layman, nothing is quite as exciting as a spectacular photo of a celestial body, taken at a higher-resolution than ever before. With all that empty space out there a lot of people just want to know, What can we see?

Read More
Published by Anna Navatsyk on Jul 14, 2015 2:03:00 AM

Announcing Product Integrations and Partner Program

VividCortex is growing, and we want you to be part of that growth! We recently released product integrations with PagerDuty, VictorOps, Slack, and Hipchat so you can notify your team of all important developments. Last month, we announced Heroku support, and our partner program has just launched.

In the words of Baron Schwartz, “Monitoring does not exist in a vacuum, and building strong relationships with other products and companies at a technical and business level is a priority for us. We always planned for this, but we didn’t plan for the demand,” he added. “The response from customers, the opensource community, and other companies has surprised us, and we’re investing even more resources than we’d planned.”

Read the full press release here.

Read More
Published by Anna Navatsyk on Jul 13, 2015 9:45:00 AM

TechSling Article: Why Database Downtime Eats Away at the Bottom Line

Speed matters. That’s why you’re willing to pay a dollar more for milk at the convenience store — it’s quicker than waiting in line at the supermarket. And that’s why you will lose business if your database slows down the customer experience.

This is especially true of e-commerce websites. Forty percent of people won’t wait more than three seconds for a mobile shopping site to load. Amazon estimates it would lose $1.6 billion a year if its web pages took one second longer to load. If consumers won’t wait one second for Amazon, they won’t wait for your database to come back online.

And that’s just the direct bottom-line impact. Poor database performance has other negative effects, but you can follow steps to avoid these problems.

Read the full article here.

Read More
Published by Anna Navatsyk on Jul 10, 2015 4:52:00 AM

Are Your Data Systems a Technology Tar Pit

Are you still running yesterday’s database? Technology and workloads have changed, so it might be time to re-examine your trusty standby. But how do you determine whether the benefits of new data storage technology are worth the cost and effort of evaluating and switching?

Perhaps you think your current data retrieval system is “good enough,” but that “just fine” system might be creating huge technical debt that could come back to haunt you. To keep up with the times, you might need to re-evaluate your database. This could result in a leaner, meaner, and more modern back end that can grow with your business and save you money.

Technology changes rapidly, and data storage technologies are no exception. Innovations such as solid-state drives are creating entirely new sets of tradeoffs and advantages. Meanwhile, the sheer amount of data being collected continues to explode. Older solutions simply cannot find good footing in this environment.

Read the full article here.

Read More
Published by Anna Navatsyk on Jul 6, 2015 3:27:00 AM

5 Database Monitoring Issues That Need Your Attention Now

As the old management adage goes, “You can’t manage what you don’t measure.” In the fast-paced world of IT, this more accurately translates to, “You can’t improve what you don’t measure.”

When it comes to your database, a continuous monitoring system provides the foundation for progress: metrics. Without a monitoring system, it’s impossible to determine whether changes actually impact the system’s performance, availability, and functionality.

Think of it like a science experiment: You wouldn’t trust research that didn’t measure the results. So why would you settle for faulty database metrics? Read the full article here.

Read More
Published by Anna Navatsyk on Jul 5, 2015 5:26:00 AM

Video Recording: Teams Are Systems Too

Earlier this year, Baron spoke at Velocity on how lessons learned from system performance can be applied to organizational structure. The talk, Teams Are Systems Too: The Theory of Constraints in Action, was largely inspired by Eli Goldratt’s book The Goal.

Read More
Published by Anna Navatsyk on Jul 1, 2015 3:37:00 AM

The Database Specialization Backlash

If you have a team whose job is to safeguard your databases as a reaction to past incidents, in an attempt to prevent them from recurring, you might have a serious bottleneck you don’t even realize.

Read More
Published by Preetam Jinka on Jun 30, 2015 4:54:00 AM

Authentication with Middleware using Siesta

We recently added a token-based authentication example to the Siesta repository that provides an excellent starting point for building authenticated APIs using our lightweight HTTP handler library for Go. This example, modeled after our own internal API services, demonstrates features and practices that we’ve developed and found useful after using Siesta in production for many months.

Read More
Published by Anna Navatsyk on Jun 29, 2015 7:51:00 AM

Success Story: VividCortex Improves Reliability at DC Energy

DC Energy is a trader in the commodities markets that relies heavily on MySQL. Since trading is time-sensitive, database management and minimal system downtime is a high priority. Prior to deploying VividCortex, DC Energy used query logs and custom tools, scripts and metrics to find and resolve problems. Internal tools did not provide a way to quickly review and assess the enormous amount of data available, and gaining insight into performance was time consuming.

Read More
Published by Anna Navatsyk on Jun 22, 2015 3:00:00 AM

Presentation: Sampling a Stream of Events with a Sketch

Last week, Baron Schwartz and Preetam Jinka presented at Monitorama. The presentation, Sampling a Stream of Events with a Sketch, addresses the challenges, available methods, and efficiencies of representative sampling. Check out the slides to view our solution, an “invented” sketch. If you would like to view the entire presentation, visit this link.

Read More
Published by Anna Navatsyk on Jun 17, 2015 5:01:00 AM

The Strategic Importance of Database Administration

If you draw a diagram of information flow and interaction amongst teams and processes in IT, you’ll probably find that although some parts of the organization are “leaf” or “edge” nodes, the people who manage the data are not. The DBAs would usually be one of the lavender circles in the chart below, not a blue circle.

Read More
Published by Anna Navatsyk on Jun 9, 2015 6:59:00 AM

Catena, High Performance Time Series Storage Webinar - Watch it Today

In this webinar, Preetam Jinka addresses a new approach to fast and efficient time series storage and indexing. He covers the unique characteristics of time series data, time series indexing, and the basics of log-structured merge (LSM) trees and B-trees. After establishing some basic concepts, he explains how Catena’s design is inspired by many of the existing systems today and why it works much better than its present alternatives.

Read More
Published by Baron Schwartz on Jun 8, 2015 6:42:00 AM

Improved User Parsing From The MySQL Protocol

This isn’t really a feature we should brag about, because it’s a bug that took us a while to figure out, but we believe in sharing the bad as well as the good. There’s a lot to learn from TCP reassembly and protocol reverse engineering!

Read More
Published by Baron Schwartz on Jun 5, 2015 11:14:00 AM

Parsing the Redis TCP Protocol

When we decided to go beyond just MySQL monitoring, we had a couple of natural next choices. The decision involved engineering effort, the likelihood we’d find MySQL-specific things in our system that would slip our schedule, alignment with existing customers, and the commercial opportunity.

Read More
Published by Anna Navatsyk on Jun 1, 2015 7:31:00 AM

VividCortex Announces Database Monitoring for MongoDB

We are excited to announce that VividCortex now supports MongoDB, a flexible and scalable NoSQL database. After announcing PostgreSQL support in January and Redis in April, our brainiacs continue to create a unified solution for diverse systems.

During the beta period, VividCortex’s MongoDB customers were impressed, as Chris Clarke, DevOps engineer at Parse.ly states: “VividCortex’s highly granular data, including detailed information about each query made against our MongoDB clusters, is invaluable. Installation was a breeze, and the system immediately spotted two problematic queries that had evaded all our other monitoring systems.”

Read the entire release here.

Read More
Published by Andres Techera on May 29, 2015 7:43:00 AM

VividCortex Adds Full-Screen Kiosk Mode

We’ve added a fullscreen (kiosk) mode to VividCortex, perfect for livestreaming a dashboard to a big monitor on your wall. Fullscreen mode uses native browser functionality to request fullscreen access, so there’s no need to try to hide your tab bar. Simply click the little icon in the top right corner of any screen:

Read More
Published by Anna Navatsyk on May 28, 2015 9:00:00 AM

VividCortex Sponsoring MongoDB Open House Hosted by Percona

Fellow lovers of tech,

We are pumped to sponsor the MongoDB Open House hosted by Percona. The event will be across the street from MongoDB World at the New York Hilton Midtown on June 1st so make plans quickly.

It is free and all are welcome.

Below are featured talks:

  • “MATH is Hard: TTL Index Configuration and Considerations,” by Kim Wilkins of Rackspace
  • “Implementing MongoDB 3.0 Storage Engine,” with Facebook’s Igor Canadi and Christian Rober of Percona
  • “Is it Fast: Measuring MongoDB Performance,” by Tim Callaghan of Acme Benchmarking
  • “MongoDB for MySQL Users,” by Percona’s Alexander Rubin
  • “Rolling out RocksDB in Production,” by Charity Majors of Facebook
  • “Percona TokuMX and Percona TokuMXSE Performance Benefits,” by Percona’s Jon Tobin

If that wealth of expertise and knowledge is not enough to convince you, there will also be free t-shirts, food, drinks and prizes.

Register here, and see you there!

Also, stay tuned for exciting news from us around the time of the event.

Read More
Published by Anna Navatsyk on May 27, 2015 5:18:00 AM

Baron Schwartz to Speak at Web Performance MeetUp in Boston

Are you trying to scale performance monitoring? Interested in the complexities of data sampling?

If you are in the Boston area Wednesday, June 10th, stop by Akamai HQ for an evening of pizza and tech.

Ben Clark, the chief architect at Wayfair, will open with a brief talk about how Wayfair is scaling performance monitoring.

Baron Schwartz will then discuss solutions VividCortex uses to effectively sample streams of data for more effective monitoring.

Click here for more details and registration.

Read More
Published by Baron Schwartz on May 26, 2015 6:31:00 AM

How We Ensure VividCortex Never Loses Data

Adrian Cockcroft really nailed it when he said that a monitoring system has to be more reliable than what it’s monitoring. I don’t mind admitting that in our first year or so, we had some troubles with losing telemetry data. Customers were never sure whether their systems were offline, the agents were down, or we were not collecting the data. Even a few seconds of missing data is glaringly obvious when you have 1-second resolution data. There’s nowhere to hide.

Read More
Published by Owen Zanzal on May 22, 2015 8:09:00 AM

Consul for Cluster Health Monitoring

If you’re not familiar with Consul, it’s what I call a cluster management tool. It’s composed of a handful of features such as “Service Discovery”, “Key Value Store”, “DNS Server”, “Health Checking”, and it’s “Data Center Aware”. It ultimately allows you to manage an infrastructure composed of many applications, dynamically configure them, route traffic to the healthy ones, and reroute traffic away from those that are not healthy.

Read More
Published by Baron Schwartz on May 21, 2015 6:41:00 AM

Updates To Our Fault Detection Algorithm

Unexpected downtime is one of your worst nightmares, but most attempts to find problems before they happen are threshold-based. Thresholds create noise, and alerts create false positives so often you may miss actual problems.

Read More
Published by Baron Schwartz on May 13, 2015 4:47:00 AM

Announcing New Free Network Analyzer Tools for MySQL and PostgreSQL

We have released two free tools that will help MySQL and PostgreSQL DBAs understand the queries their database servers execute. As you probably know, we have spent nearly 3 years building the most advanced and efficient network traffic capture and decoding tools for MySQL and PostgreSQL. With the release of these free tools, we’re placing all the power of our traffic analysis libraries in your hands.

Read More
Published by Anna Navatsyk on May 1, 2015 11:19:00 AM

Free Ticket to Sold Out Monitorama Conference

VividCortex has one ticket to the SOLD OUT MONITORAMA 2015 conference, and we want to give it to someone deserving. Fill out the form for your chance to win a ticket for yourself or someone else, and enjoy your journey to better monitoring!

Read More
Published by Anna Navatsyk on Apr 30, 2015 8:05:00 AM

Introducing the Power of Notifications

Since VividCortex launched its database monitoring platform, many users have requested some type of notification system. To address this, we took particular care in building a feature that proactively warns users of potential problems without sending multiple false alerts.

Read More
Published by Anna Navatsyk on Apr 28, 2015 6:37:00 AM

Improvements in VividCortex's Query Digesting Algorithm

At VividCortex, the algorithm our agents use to digest queries needs to be fast and reliable. Accurately digesting queries into logical groups is important to providing a useful view of data in the UI - digest too few or too many and it becomes difficult to understand what the Top-Queries view is showing.

Though our algorithm is sophisticated, one of our PostgreSQL clients experienced an issue with query digestion. They had many queries of the form “select a,b,c from table where c = any(values (1), (2), (3)…)” which were not being collapsed correctly.

We recently improved our digest code to correctly digest queries of this form.

In both MySQL and PostgreSQL, “SELECT * FROM t WHERE a IN (SELECT aa FROM tt)” was incorrectly collapsed to SELECT * FROM t WHERE a IN (?). Now, the digest is the lowercased original query text, as there is nothing to collapse or conceal.

These are improvements that help our clients have clearer visibility into their database activity. We are always listening to customer requests, and our support brainiacs are constantly working to improve visibility and service. If you have not given us a try, sign up for a free trial today.

Read More
Published by Anna Navatsyk on Apr 21, 2015 7:47:00 AM

VividCortex Receives the MySQL Application of the Year Award

Last week at Percona Live, VividCortex received the MySQL community award for 2015 Application of the Year.

Baron Schwartz, our founder and CEO, is quoted, “It’s an honor to receive this award from the MySQL community. We aim to raise the bar for database monitoring the same way MySQL has raised the bar for open source databases. Our product would not be at this point without the dedication and help of our employees, friends, customers and investors. We thank them for their support and look forward to many years of mutually beneficial relationships. On a personal note, as a previous recipient of Community Member of the Year award, and having dedicated the last decade of my life to the MySQL community, this is deeply meaningful to me.”

If you have not yet tried VividCortex, sign up for a free trial to get unprecedented insights into your databases.

Read More
Published by Anna Navatsyk on Apr 13, 2015 11:30:00 AM

Potential Perils of Using Google Calendar and Hangouts

Lots of businesses are using Google Apps, and with good reason. Google Calendar streamlines schedule coordination and is especially vital to use with sales prospects because the opportunity cost of getting a timezone wrong or having a prospect fail to show up is too high.

However, the Google approach still has a couple problems. First, Google Hangout is possibly the single worst way to have a call due to all kinds of plugin and permission issues. If the user could be invited with an email address from a secondary account, they may be unable to join. There are also hardware/audio/video issues, echoes, poor connections, etc. In short, Google Hangouts is a great way to completely fail the call.

If you choose to use another service, invitees can be really confused when they get an invitation to a GoTo Meeting event because Google auto-adds hangout links to calendar invitations. Which are they supposed to open?

There is a solution. Instead of Hangouts, use phone, join.me, or gotomeeting – something that has good audio quality, allows direct dial-in, and doesn’t require any plugins to be installed by default.

Still use Google Calendar invitations, but disable auto-adding of hangouts. That way, you can still manually add if desired, but it is not automatic.

This requires the domain admin to disable it on a domain-wide basis. From admin.google.com, go to Apps, find Calendar, select Sharing Settings, and uncheck the “Automatically add video calls” checkbox at the bottom. For more details see https://support.google.com/a/answer/4362302.

Read More
Published by Anna Navatsyk on Apr 6, 2015 8:51:00 AM

VividCortex php-recaptcha-validator

We recently updated our angular-recaptcha to use the new Google’s simplified captcha.

Validations must be done server side, so one of our brainiacs created php-recaptcha-validator as a companion to angular-recaptcha. You can do this in whatever language you prefer; this is an option for PHP and is MIT licensed.

The README file contains the simple steps to install and use it.

To install, simply add your project requirement using composer

You will need to instantiate a new validator with your secret key provided by Google.

$validator = new Validator($secret);

Then, you can move on to validating responses. The response and the client’s IP are required.

// Returns TRUE or FALSE
$result = $validator->validate($response, $clientIp);

As always, we welcome comments and contributions.

Read More
Published by Anna Navatsyk on Apr 3, 2015 3:29:00 AM

Shortcut Your Way Through VividCortex

Appreciate efficiency? So do we. Your fingers are moving all day, and we want to make your navigation experience as streamlined as possible. When you are using our app, hit “?” to view the list of shortcuts below.

Read More
Published by Anna Navatsyk on Apr 2, 2015 7:33:00 AM

VividCortex Announces Database Monitoring for Redis

We are excited to announce that VividCortex now supports Redis, a popular open source database. After announcing PostgreSQL support in January, our brainiacs continue to create a unified solution for diverse systems.

Baron Schwartz explains the product development further. “Redis is an important database for VividCortex, both strategically and commercially, due to its incredible popularity and our own usage. We rely on Redis to help analyze the massive amounts of time-series data we receive from agents running on customer systems. Although it is a high-performance and reliable system that generally just works, that doesn’t mean it can’t be improved. We were able to significantly optimize our usage of Redis with the network-traffic visibility built into our agents. Most of our customers also use Redis, and we are delighted to offer our advanced solution to the general public now. The visibility provided into Redis’s performance, as well as the app’s usage of it, is simply unparalleled in the market.”

Read the entire release here.

Read More
Published by Anna Navatsyk on Apr 1, 2015 8:17:00 AM

Introducing VividCortex Google Glass App

VividCortex is thrilled to announce the latest in personalizing your database monitoring experience. System stall? On the go? Look no further than directly in front of your face. Our Google Glass app allows you to see unprecedented query detail even when away from your computer. Save precious seconds as you interactively scroll through features to solve your database problems while in the store, out for coffee or home for dinner.

Read More
Published by Anna Navatsyk on Mar 25, 2015 7:49:00 AM

An Inside Look: Building Database Driven Apps with Go

Have you downloaded our eBook, Building Database Driven Apps with Go? It's the best resource to quickly learn how to connect a database from Go programs.

This excerpt presents a quick introduction to the major functionality of database/sql in the form of a fully functioning Go program.

Your First database/sql Program

Before you begin, ensure you have access to a MySQL database, as we’ll use MySQL for the example. If you don’t have an instance of MySQL that’s appropriate for testing, you can get one in seconds with the MySQL Sandbox utility.

Create a new Go source file, hello_mysql.go, with the following source code (download). You may need to adjust the connection parameters as needed to connect to your testing database. Note also that the example assumes the default test database exists and your user has rights to it:

 

 

Run your new Go program with go run hello_mysql.go. It’s safe to run it multiple times. As you do, you should see output like the following as it continues to insert rows into the table:

 

Congratulations! You’ve written your first program to interact with a MySQL server using Go, and here are a few highlights:

  • You imported database/sql and loaded a driver for MySQL.

  • You created a sql.DB with a call to sql.Open(), passing the driver name and the connection string.

  • You used Exec() to create a table and insert a row, then inspect the results.

  • You used Query() to select the rows from the table, rows.Next() to iterate over them, and rows.Scan() to copy columns from the current row into variables.

  • You used .Close() to clean up resources when you finished with them.

What might surprise you is that the code you’ve just run is not some kind of overly simplified, silly example. It is very similar to the code you’ll use in production systems under high load, including error handling. Much of this code is discussed in further sections of this book, available for download here.

Read More
Published by Anna Navatsyk on Mar 16, 2015 5:55:00 AM

Three Ways To Generate Technical Content

Your engineers are your best source of content, as they are doing the most impactful and relevant work. However, they’re busy and their time is more economically spent writing code. Plus, they likely would rather write code than blog posts. That does not mean you have to leave all that knowledge unshared, however. You just have to get a bit creative to extract it.

Nudge Engineers, But Don’t Push

Although you want to get people out of their comfort zone to stimulate ideas, don’t ask people to do what they’re not good at. “Content generation” can be intimidating, so frame it in a different light. Ghost-write for them so they feel like all they’re doing is teaching you about a subject, playing to both of your strengths.

Notice The Small Things

The hardest part is generating ideas. That’s because you shouldn’t be creating ideas; you should be noticing what’s already there. Every engineer works on three things a day that could help the community, but it doesn’t occur to them. Notice what they’re doing and jot down that topic as an idea for a blog post. You can later ask them some questions and build an outline from the answers.

Focus On Being Useful

The best content is that which answers questions and solves customer problems rather than merely selling your product or self-promoting. Help people learn how to solve their problems as a goal. If it happens to be the same problem your product is solving, it will flow naturally.

Our team has written plenty of blog posts focused on problems we face with the goal of benefitting the community. Below are just a few:

Smoke-testing with a sanity check tool by Preetam Jinka

Using Netlink to optimize socket statistics by Fernando Papa

Query samples, explained by John Potocny

Grunt plugin for circleci by Ismael Ambrosi

Read More
Published by Anna Navatsyk on Mar 12, 2015 8:14:00 AM

Case Study: VividCortex Improves Time to Market for NetProspex

NetProspex is a B2B platform that enhances lead generation by separating the information that matters and enabling marketers to better segment and target their audience. Managing the massive amount of incoming data is cumbersome for one DBA.

In the past, NetProspex used tools like Nagios and MySQL Enterprise Monitor (MEM), but both fell short of the company’s monitoring needs. Hiring another DBA was both difficult and expensive, so Josh Prunier, DBA at NetProspex, began evaluating alternative solutions. VividCortex’s MySQL-specific query intelligence was better suited.

VividCortex made an immediate positive impact on Josh’s workload and overall operations by improving insight into query and server behavior and reducing overhead costs. The ability to compare and analyze queries allows Josh to proactively examine queries that are not written efficiently and take actions such as tweaking an index or the query syntax. The deep detail exposed via VividCortex’s TCP traffic capture and advanced statistical analysis immediately surfaces “bad” queries that are taking a long time, using a lot of IO resources or not using indexes.

According to Josh, “VividCortex is a database performance monitoring solution for enterprise systems. It makes a DBA’s and developers’ job easier.”

VividCortex has resulted in faster deploys to improve time-to-market, reduced resource costs and generally reduced headaches for DBAs and developers, making everyone happier.

The full case study is available for download here. If you think this level of analysis could benefit your company, learn more or start your free trial today!

Read More
Published by Anna Navatsyk on Mar 10, 2015 11:10:00 AM

Five Steps to Great Trade Show Demos

Spring is approaching, and with it, the season of sponsorships. For vendors, trade shows offer a unique opportunity to gain valuable insight regarding market needs, engage potential customers, and show off their product to hundreds. In order to maximize this opportunity, your team needs to enter with a plan. Here are five tips we keep front of mind to put our best foot forward when demoing VividCortex at trade shows.

Read More
Published by Baron Schwartz on Mar 9, 2015 10:13:00 AM

Monitor the Source of MySQL and PostgreSQL Database Load

VividCortex now computes metrics about each client connection’s host (where the remote end of the connection originates) in Top Queries, so you can monitor MySQL and PostgreSQL activity per remote host. By changing the Rank menu from Queries to Hosts, you can see a breakdown of activity in the familiar format, all with 1 second resolution.

Read More
Published by Baron Schwartz on Mar 7, 2015 5:11:00 AM

A Handy GitHub Backup Script

We use GitHub to host all of our source code at VividCortex. We use the GitHub API in a very simple way to ensure we’ve backed up all of the source code, because we add new repositories often.

Read More
Published by Baron Schwartz on Mar 5, 2015 4:43:00 AM

Analyzing Related Metrics With VividCortex

Last week we announced our new query and metric listing and detail pages, which provide deep drilldown into individual queries and metrics for exploration and analysis. Today I want to show you one of the new features included in the Metric detail page. We have used it in a variety of scenarios for customers and for our internal analysis. We’ll cover some of the usage cases and success stories in future blog posts; here I’ll just give an overview of the capability.

Read More
Published by Baron Schwartz on Mar 5, 2015 3:13:00 AM

Are Scatterplots The Killer Data Visualization Tool You're Not Using?

At VividCortex, we deal with a lot of data, day in and day out. It’s not just our customer data: we actually consume a lot of data and reports produced by others as well. For example, there are reports in Salesforce’s CRM, various analytics solutions we use, industry reports we read, and on and on. Speaking personally, I probably study at least 10 charts a day in depth to try to understand their meaning. And it’s only growing as our business and activity grows.

Read More
Published by Anna Navatsyk on Mar 3, 2015 6:22:00 AM

March 17 Webinar: How Indexes Work in MySQL

MySQL offers a few different types of indexes and uses them in a variety of ways. There’s a lot to know about the various kinds of indexes and how they interact with the storage engines, and it’s all very important for query optimization. A few examples are listed below:

  • The “leftmost prefix rule”

  • Clustered primary indexes versus secondary indexes

  • B-Tree and hash

  • New types of indexes such as LSM and Fractal Trees

  • Newer features in the query optimizer and executor in MySQL 5.6 and 5.7

You will leave this webinar with a better understanding of how MySQL and its storage engines use indexes to speed up queries, and how you can improve query performance with basic and advanced index optimizations. Please register here, and no worries if you cannot attend the webinar. We will send you a recording.

Read More
Published by Baron Schwartz on Mar 2, 2015 5:34:00 AM

Temp Tables, Filesorts, UTF8, VARCHAR, and Schema Design in MySQL

The other day we were doing some peer review on a schema change at VividCortex and the topic of character set, column length, and the potential downsides of using utf8 came up. I knew from experience that there are some hidden gotchas with this, and usually I’ll just say a couple words and send a link with further reading. But Google turned up nothing, so I am writing this post to try to help fix that.

Read More
Published by John Potocny on Feb 26, 2015 3:20:00 AM

See Connection Latency with VividCortex

Recently, one of our customers reached out to us about a problem they were having trouble diagnosing - they noticed that every once in a while, it took their MySQL server a few seconds to initialize a client connection. They had figured out that the issue was a missing hosts entry for one of their clients, but VividCortex didn’t have any measurements to show this issue to them.

Read More
Published by John Thomas on Feb 25, 2015 7:57:00 AM

Introducing Query and Metric Pages

We recently deployed multiple new pages at VividCortex providing detailed query and metric information. To access this level of information previously, you had to utilize Top Queries, one of our most popular tools. While the power of Top Queries lies in the ability to rank queries by specific criteria, these new pages allow users to easily filter and see detailed information about any query or metric.

Read More
Published by Baron Schwartz on Feb 24, 2015 3:16:00 AM

Schemaless Databases Don't Exist

There’s no such thing as a schemaless database. I know, lots of people want a schemaless database, and lots of companies are promoting their products as schemaless DBMSs. And schemaless DBMSs exist. But schemaless databases are mythical beasts because there is always a schema somewhere. Usually in multiple places, which I will later claim is what causes grief.

Read More
Published by Anna Navatsyk on Feb 18, 2015 8:11:00 AM

Case Study: VividCortex Helps WebAssign Prepare for High Volume Season

WebAssign develops higher education online instructional tools for faculty and students, and more than one million students at over 2,300 educational institutions use WebAssign each year. Last year, WebAssign experienced a large outage that impacted their ability to provide timely online grading and assessments; they took this opportunity to look for new tools that would improve insights into their database activities and ensure service level requirements.

“MEM for MySQL wasn’t able to provide per-query metrics or fault diagnosis, which is a level of detail that we value,” said Valerie Parham, DBA, WebAssign. “Instead, releases would appear fine, but then a problem would pop up and we’d have to do a hot-fix so it didn’t turn into an outage, which placed undue stress on everyone.”

WebAssign looked at a number of solutions, including Percona Cloud Tools, but none offered the same level of full query insight and predictive capability of VividCortex. “VividCortex is a monitoring system that gives us full insight into our database - at the system level, the query level, and the database performance level,” said Parham. “Plus, it’s very usable even by non-technical people.” The level of detail is unmatched by other tools and enables smooth problem identification and resolution.

This season, WebAssign was well prepared because they were able to detect and fix problems well in advance.

The full case study is available for download here. If you think this level of analysis could benefit your company, start your free trial today!

Read More
Published by VividCortex on Feb 12, 2015 6:15:00 AM

Brainiac Corner with Charity Majors

The Brainiac Corner is a format where we talk with some of the smartest minds in the system, database, devops, and IT world. If you have opinions on pirates, or anything else related, please don’t hesitate to contact us. Today, we interview Charity Majors, the production engineering manager for Parse at Facebook. Prior to the acquisition, she built Parse’s infrastructure. Shes hates/loves/hates databases and loves whiskey.

Read More
Published by Baron Schwartz on Feb 10, 2015 6:26:00 AM

How VividCortex Measures Queries

We often get questions about how we measure queries inside MySQL (and now PostgreSQL too). Many of these questions reveal assumptions about the metrics we provide and our methods for obtaining them. I thought it would be both interesting and helpful to clarify how we really do this. This article may get a bit dense, but by the end you will have a better sense of our database management platform’s unique approach.

Read More
Published by Anna Navatsyk on Feb 9, 2015 9:14:00 AM

Free Webinar! What Should I Monitor, and How Should I Do It?

Monitoring tools offer alerts based on thresholds and time-series charts of status counters, but they do not tell us what we should monitor. On Tuesday, February 10th at 2 PM EST, Baron Schwartz will discuss the guiding principles needed to answer these questions.

The following questions will be answered:

  • What status counters from MySQL are central and core, and which are peripheral?
  • What is the meaning of MySQL status metrics?
  • Which subsystems inside MySQL are the most common causes of problems in production?
  • What is the unit of work-getting-done in MySQL, and how can you measure it?
  • Which open-source tools do a good job at monitoring in the way we recommend at VividCortex?
  • Which new and/or popular open-source tools should you evaluate when choosing a solution?

To find out more information and register, click here. If you cannot attend, we will send you a link to the recording.

About the speaker

Baron is one of the world’s leading experts on MySQL, and has helped build and scale some of the largest web, social, gaming, and mobile properties. His award-winning tools are used by tens of thousands of large MySQL deployments, including Facebook and Twitter. His book High Performance MySQL is widely acknowledged as the definitive reference for MySQL. As an author, consultant, speaker, and conference organizer, Baron has deep connections in the MySQL and DevOps communities and beyond. Before founding VividCortex, Baron led sales, marketing, consulting, support, and software engineering at Percona, the leading MySQL consulting and support company.

Read More
Published by Anna Navatsyk on Feb 5, 2015 7:21:00 AM

In Case You Missed It - Stackscope Industry Webinar

This is a must see for anyone in the ever-transforming technology industry. Join Adrian Cockcroft, Jason Dixon, Nathen Harvey and Baron Schwartz discuss the Internet of Things. Some highlights:

  • How do we reconcile the containerization dissonance?

  • What are the advantages, scaling hurdles and security implications of Docker?

  • What monitoring methods will ultimately reign supreme, and what tools are ahead of the curve?

  • What are the paths toward DevOps, and will everyone eventually adapt?

  • How do companies like Chef, Librato and NetFlix use containerization, monitoring and DevOps?

  • What is the future of database as a service and the next big programming language?

  • The 2020 landscape

Over the coming weeks, we will be posting bits and pieces, but you can register for a full recording here. And you should!

Read More
Published by Anna Navatsyk on Feb 3, 2015 8:35:00 AM

VividCortex Releases Developer Service Tier for the Lean Startup

Today we announced the release of VividCortex Developer edition. You can find the full press release here.

VividCortex’s new Developer service tier makes industry-leading query-level visibility available for no charge.

Baron Schwartz is quoted: “As a technology startup, VividCortex appreciates the value of moving fast. When a small team of smart developers have just founded a business and are searching for product/market fit with a minimal viable product, an excellent tool for database performance management is a must. The last thing you want to do is spend time on burdensome tasks like slow query log analysis. We’re big proponents of the lean startup mindset here at VividCortex, and we’ve benefited greatly from the opensource community, so this is one way we can say thanks.”

To keep costs low and make it possible to give forward to the next generation of makers and hackers, some of the more data-intensive features in the new Developer edition service tier are less robust. Of course, our Professional and Enterprise tiers are full-featured, adaptable and ready to scale.

Visit our pricing page to view our three tiers, decide which is right for you and begin your 14-day free trial!

Thumbnail Cred

Read More
Published by Anna Navatsyk on Jan 29, 2015 8:01:00 AM

Case Study: VividCortex Gives Dyn Predictive Smarts and Analysis

Dyn delivers Internet Performance solutions to optimize clients’ Internet properties. Data is their business, so it is crucial their systems are running effectively. Prior to VividCortex, Dyn used a variety of tools including Graphite, Cacti and Munin to analyze health and performance. Despite custom reports and graphs, Dyn was left with data that lacked predictive smarts and was difficult to interpret.

Dyn evaluated other products like Anemometer, Percona Toolkit, MONyog and New Relic but ultimately chose VividCortex for its depth and consolidation of insight. Features such as “compare queries” present a clear perception of change, a feature not common in competitor’s tools. VividCortex analyzes latency across disc IO, the network and query performance to easily pinpoint the source of problems.

Tim Chadwick, Principal Engineer of Infrastructure says it best: “At this point, it is considered a must have, and everyone on the team uses it to see the effects of system changes. Beyond that, the functional interface makes it easy to share findings across departments and communicate needs effectively.”

The full case study is available for download here. If you think this level of analysis could benefit your company, start your free trial today! You will have immediate access to our app and support team, who is, in the words of Tim, “incredibly helpful and responsive.”

Read More
Published by Baron Schwartz on Jan 27, 2015 6:18:00 AM

How VividCortex's Agents Manage Logs

It can be scary to run agents on your critical servers. Misbehaving agent software can cause harm, including pegging your CPUs, filling up your disks, or eating all your memory and making your server swap. Fortunately, at VividCortex we have many years of experience with these problems and we designed our agents to avoid them from day one.

Our agents are self-limiting in every aspect of resource consumption, including log files. Unlike some software you may have experience with, VividCortex’s agents won’t fill your disks with logs or temp files, nor will they cause a lot of I/O. We ensure this through a variety of measures, but there are three basic techniques I’d like to mention here.

  1. The agents don’t generate or keep spools of data. We don’t spool metrics to disk, for example, nor do we create temporary files or other caches. Our metrics aggregator agent will retry failed transmissions up to 5 times by default, but that represents a very minimal amount of memory and doesn’t impact disk at all.
  2. We only log minimal information. It’s much better to avoid logging than to deal with the consequences of verbose logging. In normal operation, VividCortex’s agents will only log a periodic indication that they’re successfully communicating with our APIs. It’s basically a liveness heartbeat.
  3. The log messages that the agents do write won’t grow indefinitely. They essentially have a built-in logrotate mechanism. They will truncate and expire their own logs. This protects you against filling up your disks.

This last bit may sound silly – why don’t we just use standard logrotate, instead? – but it’s actually very important. My experience as a consultant taught me that relying on system-provided functionality is a path to madness, because facilities like logrotate have bugs and undesired behaviors in a variety of circumstances. Furthermore, many of these might be triggered by nonstandard system configurations out of our ability to anticipate, detect, and control. For these reasons we ship a known good, clean implementation of everything we need (except for certain core system libraries), baked right into our agents.

Our agents are fully configurable, so all of the above are subject to tweaking and tuning if customers want. However, out of the box, our agents are configured to “just work” and be very respectful of system resources. One of the fastest ways to kill a system is to fill up its disk or cause a lot of I/O. High-performance servers, in particular, often run very close to a performance cliff, a delicate state where just a little bad behavior can cause the whole thing to tip over and die or freeze.

And how do we know that our agents don’t cause these problems? Well, because we capture per-process performance data in 1-second resolution – including CPU, memory, network, disk and more – if there’s a problem, we can see it. And because our Adaptive Fault Detection technology detects micro-stalls as short as 1 second long, we also know when a badly-behaved program impacts others. And we do see this, all the time – with other software, and sometimes even with other monitoring software. But not with our agents.

Sign up for your free trial and experience the benefits of VividCortex for yourself!

Read More
Published by Baron Schwartz on Jan 23, 2015 11:28:00 AM

Distributed and Diverse: The New Reality of Modern Data Persistence

We create applications in an age of simple, powerful, flexible databases that do magic for us. There’s a large variety of modern databases that supply just what’s needed for lots of use cases, so we can pick the right tool for the job. We’ve never had it better, right? So why is “it’s the database again” still a sufficient explanation for a lot of outages and performance problems?

Read More
Published by Baron Schwartz on Jan 21, 2015 3:18:00 AM

Securing JSON APIs with Wrapper Objects

At VividCortex, security is a top priority. Leading companies such as Zappos, Dyn, and Etsy use our cloud-based database performance management service to monitor MySQL in production. We have designed for performance, isolation, and security from the start.

Even small decisions can make a big difference. One of those micro-decisions is making all of our APIs return a top-level object in JSON, never a top-level array. This is to avoid an old, obscure, unlikely, but still possible JSON security vulnerability.

If you have never heard of JSON security vulnerabilities, you should go read Anatomy of a subtle JSON vulnerability before continuing.

Now that you’ve read that, you know a lot more than most people about JSON and security! Although modern browsers have fixed the underlying vulnerability, older browsers are still in use, and so although it’s not as convenient, we think it’s still important not to leave this potential hole open to exploit.

What does this look like in practice? It simply means that APIs that return lists of objects wrap the list in a top-level object. For example, suppose you have a Person data type in your API. If you GET a single Person, you might end up with the following JSON response:

{ "name": "John Smith" }

All’s well so far. But what if you want a list instead of a single one? The most intuitive way might be as follows:

[{"name": "John Smith"}, {"name": "Jane Smith"}]

But that’s exactly the situation we need to avoid. As a result, all of our APIs at VividCortex use a top-level object for lists such as this, with a single data property:

{"data": [{"name": "John Smith"}, {"name": "Jane Smith"}]}

Some people object to this; it seems inherently wrong to them. Our view is that security trumps elegance, and it’s not a big deal for consumers of the API to look for a nested list instead of a top-level list. Developing a large-scale service-oriented application involves making lots of decisions, many of which include tradeoffs. Here are a few resources you may find interesting:

What do you think of this convention and the underlying security concerns? Leave your suggestions and thoughts in the comments!

Thumbnail Cred

Read More
Published by Anna Navatsyk on Jan 20, 2015 8:09:00 AM

Case Study: VividCortex Helps PayLease Manage Massive Amounts of Data

PayLease is the leading online payment processor for property managers and HOAs. The company serves thousands of property management clients across the United States and was seeking a solution to help them better manage their massive amount of data and minimize slow periods.

Before VividCortex, they had to sort through nearly 10 gigabytes of data each time there was an issue, a cumbersome process that often proved ineffective. PayLease also wanted a tool that would better equip them to prevent future issues. They needed a predictive solution to flag issues that could become high priority down the road.

PayLease chose VividCortex because of the intuitive interface, level of granularity and functionality. They saw the difference immediately. Todd Newell, the Database Engineer, “I look at top queries on a daily basis. I recognize what is there, what should be there, and am able to quickly pull something out that is garnering way too many resources and re-architect it. The top queries view shows which queries were running, each version of those queries, the number of executions, and the amount of resources consumed. On an investigation level, this is extremely helpful.”

Now, they are able to recognize problems at the time of code release and immediately act. The granular detail of VividCortex allows them to see specific queries having issues and eases the pain of diagnosis. What typically took 3 – 4 hours and yielded uncertain results now renders actionable results immediately. The usability of the interface allows them to share this information with team member and management to prevents future issues and make their needs known. Beyond that, PayLease is no longer having to store and search through slow query logs, reducing their query storage from between 35 GB and 100 GB per day to less than 1 GB.

The full case study is available for download here. If you think this level of analysis could benefit your company, start your free trial today! As Kevin Palmer, Senior Systems and Software Engineer at PayLease stated, “It has features I never knew I needed like the ability to compare queries over time. It is becoming more and more integral in preventing and finding issues.”

To view the archives of case studies, visit here.

Read More
Published by Baron Schwartz on Jan 19, 2015 5:00:00 AM

Go's Connection Pool, Retries, and Timeouts

This is a story of intermittent 500 Internal Server errors from APIs, that ended up being caused by a hardcoded constant in Go’s database/sql package. I’ll mostly spare you the long-winded story, and get right to the problem and what we found to be the cause.

Read More
Published by Anna Navatsyk on Jan 13, 2015 8:34:00 AM

VividCortex Brings Performance Intelligence to PostgreSQL

Today we announced the release of VividCortex for PostgreSQL. You can find the full press release here.

It has always been our goal to bring users deeper analysis of everything in the system through a unified interface across a diversity of platforms and technologies. Since launching last year, we have given our customers an unparalleled view under the hood of their MySQL production database servers, allowing developers, managers and executives to see exactly what is happening.

That same insight and analysis is now available for PostgreSQL, the most advanced open-source relational database system. The highly scalable database manages terabytes of data in many production deployments. Such demanding uses require proper performance management for maximum efficiency, and VividCortex is the only product on the market that offers the fine-grained detail needed.

Tim Chadwick, Principal Engineer of Infrastructure at Dyn, explains how VividCortex is used to improve overall system performance. “Our goal is to have a single tool for assessing the health and performance of our data plane. As an OSS supporter leveraging the strengths of MySQL and PostgreSQL, VividCortex allows us to accomplish this and gives us more insight than any other tool.”

This is neither the first nor the last database we will service, and we are excited for what is to come. Stay tuned for ongoing improvements and new features.

If you are interested in the benefits, a free trial of VividCortex for PostgreSQL is available through a simple signup. Installation takes minutes and requires no server restarts or configuration changes.

Read More
Published by Mario Sukkar on Jan 8, 2015 3:26:00 AM

Monitor MySQL Database Users with VividCortex

VividCortex now supports per-user metrics in Top Queries, so you can monitor MySQL database user activity in detail. By changing the Rank menu from Queries to Users you can now see a breakdown of user activity in the familiar format, all with 1 second resolution. Here “user” means the database user account that the application uses to access the database.

Read More
Published by Anna Navatsyk on Jan 6, 2015 4:13:00 AM

StackScope Webinar Series: Industry Roundtable

It’s a new year, and that means there are new challenges in the ever shifting technology landscape. Begin 2015 ahead of the curve. On January 22nd, StackScope will host its first webinar, an industry roundtable. Four industry leaders will discuss topics including the Cloud, Big Data, Microservices, Continuous Delivery, Containerization, and the Internet of Things. They will address questions from the future of DevOps to the impact of Amazon Aurora and everything in between. You do not want to miss the opportunity to participate with these tech vets:

  • Adrian Cockcroft, World-renowned Cloud Computing and Performance Optimization expert, author, and Technology Fellow for the investment firm Battery Ventures.

  • Jason Dixon, the creator of the worldwide Monitorama conference series and Director of Product Solutions at Librato, a platform for monitoring and alerting,

  • Nathen Harvey, DevOps master and community leader at cloud management software company, Chef, and the organizer of DevOpsDC, and

  • Baron Schwartz, founder and CEO of VividCortex, one of the world’s leading experts on database performance, and author of High Performance MySQL

To read more about the participants and register, click here. We also would love topic suggestions ahead of time, tweeted to #stackscope.

It is certain to tantalize your technology tastebuds, and the experience of the speakers will help you distinguish trends from shifting reality and equip you to thrive in tomorrow’s landscape. So sign up, invite your colleagues, grab a cup of coffee and enjoy!

Read More
Published by Anna Navatsyk on Jan 5, 2015 10:19:00 AM

Enhanced Support for Per-Process Metrics on FreeBSD

VividCortex has always tracked process activity to help monitor and explain database performance, and recent enhancements to the Top Processes feature provide even more clarity. Often, processes impact one another in unexpected ways, and monitoring both MySQL and system resource utilization at high resolution is the only way to pinpoint those instances.

Read More
Published by Baron Schwartz on Dec 18, 2014 10:09:00 AM

Monitor MySQL Performance Interactively With VividCortex

If you’re monitoring MySQL performance on dozens or hundreds of servers, chances are you have a “rainbow chart” – a time-series chart with hundreds of tiny area graphs stacked on top of each other, crawling imperceptibly one pixel at a time across the big-screen monitor in your office. The trouble with these charts is they’re hard to see. It takes many minutes for enough new pixels to display after a change. In the meantime you can’t see the change clearly.

Read More
Published by Baron Schwartz on Dec 17, 2014 11:03:00 AM

December Tip of the Month: Find Queries that Lack Indexes

It’s an ancient tradition amongst MySQL DBAs to analyze slow query log files and sort the results by the so-called “index ratio” to find badly indexed queries. The “index ratio,” in this case, is a heuristic derived from the ratio of rows read to rows returned. The thinking goes that if a query is reading lots of rows and returning only a few, it might be missing an index.

Read More
Published by Anna Navatsyk on Dec 10, 2014 8:42:00 AM

VividCortex Presents StackScope

Today we announced StackScope, a new industry colloquium for IT technologists. The amount of available technology is overwhelming and continuing to grow. The goal of Stackscope is to help you discern the hype and buzzwords from what is necessary to remain on the cutting edge.

The first webinar will be held January 22nd at 2 PM EST. The roundtable will include Adrian Cockcroft, Jason Dixon, Nathen Harvey, and Baron Schwartz. These thought leaders will discuss Cloud, Big Data, Continuous Delivery and the Internet of Things.

In the words of Adrian Cockcroft, Technology Partner at Battery Ventures,

“We make choices in our personal and business lives based on assumptions about what is possible and the relative costs and risks of different options. Over time our assumptions drift off target as reality changes, and the optimizations we make in life and work may become invalidated. Tracking the underlying trends and making different optimizations is one of the ways that disruptive companies can create new business models.”

Register to attend the event here, invite your colleagues and peers, and stay ahead of technology in 2015.

Read More
Published by Baron Schwartz on Dec 8, 2014 4:47:00 AM

Global Counters And Average Patient Temperature

Our recent blog post on finding wasted prepare/execute/close cycles showed how per-statement analysis can find individual queries that are not being repeatedly executed after being prepared. To recap, we found a number of statements in our own application that were being prepared, executed, and then never re-executed. This means 3 network round-trips instead of just executing the statement directly, which takes only 1 round-trip. By fixing this, we materially decreased load on our servers and improved latency for the service that issued the query.

Read More
Published by Baron Schwartz on Dec 4, 2014 10:01:00 AM

Why VividCortex is Cheaper Than DIY Open-Source

Talk to someone who runs monitoring systems at a company with more than a few servers and you’ll quickly find out that scaling their monitoring systems is far from a minor concern. It’s often a serious problem. The three primary costs of self-hosting an open-source monitoring system, according to our survey of hundreds of Operations staff at large companies, are:

Read More
Published by Anna Navatsyk on Dec 2, 2014 6:15:00 AM

Free Webinar! Building a Time-series Database with MySQL

Relational databases aren’t very well suited to time-series data in some ways, but MySQL is powerful and flexible. At VividCortex, we have found that it’s a good storage engine to support a large-scale, high-velocity time-series database in the AWS cloud.

Join us next Tuesday, December 9, at 2 PM EST (7 PM GMT) as Baron Schwartz, our CEO and Founder discusses our unique time-series data requirements. He will explain the following:

  • How we built a solution using MySQL and additional components such as Redis
  • Why we needed more than just MySQL to meet the requirements
  • The good and bad aspects of our architecture
  • Performance statistics
  • Thoughts for the future of our time-series data architecture

You will leave the talk with a greater understanding of MySQL’s capabilities related to time-series data. Please register here, and no worries if you cannot attend the webinar. We will send you a recording.

About the speaker

Baron is one of the world’s leading experts on MySQL, and has helped build and scale some of the largest web, social, gaming, and mobile properties. His award-winning tools are used by tens of thousands of large MySQL deployments, including Facebook and Twitter. His book High Performance MySQL is widely acknowledged as the definitive reference for MySQL. As an author, consultant, speaker, and conference organizer, Baron has deep connections in the MySQL and DevOps communities and beyond. Before founding VividCortex, Baron led sales, marketing, consulting, support, and software engineering at Percona, the leading MySQL consulting and support company.

Read More
Published by Baron Schwartz on Dec 1, 2014 2:57:00 AM

Running VividCortex Via An HTTP Proxy

Many of VividCortex’s customers have strict policies about Internet access from and to their database servers. As a longtime consultant who worked on many servers that lacked a direct connection to the Internet, I anticipated this. Since the beginning, VividCortex has been designed to work in deployment scenarios without requiring direct Internet access, and requiring no inbound access at all.

Read More
Published by Natasha Richter on Nov 24, 2014 7:44:00 AM

Leading the Way: VividCortex at the Leading Edge Forum Study Tour

A few weeks ago, our founder Baron Schwartz shared the unique opportunity of presenting alongside Adrian Cockcroft, one of Cloud Computing’s leading thinkers, at the Leading Edge Forum’s Study Tour. The Silicon Valley-based tour was attended by some of the brightest and most forward-thinking minds in business, and it was an honor to speak to the group. Forty European CIOs and leaders from prestigious businesses and organizations gathered to hear about how the world’s movers and shakers approach innovation—and how they can anticipate and leverage emerging technology changes to their competitive advantage.

Read More
Published by John Potocny on Nov 19, 2014 4:30:00 AM

Analyzing Prepared Statement Performance With VividCortex

Optimizing MySQL performance requires the ability to inspect production query traffic. If you’re not seeing your application’s production workload, you’re missing a vital part of the picture. In particular, there are lots of performance optimizations your systems might be doing that you’re not aware of. One of these is using prepared statements for queries.

Read More
Published by Baron Schwartz on Nov 17, 2014 7:19:00 AM

View Per-Process I/O and More With VividCortex

What can explain mysterious slowdowns in database performance? Sometimes it’s not the database or the queries it’s running, but the non-database activity on the server. One of the most important types of activities to analyze is processes. That’s why VividCortex tracks process activity, including per-process CPU, memory, and I/O, in 1-second detail. It’s like a historical view of top, and you can view it across your entire environment in a single screen, then drill down to individual servers and programs. If you see interesting regions you can click and drag with the mouse to zoom in and correlate. You can rank and sort by any of the per-process metrics we capture, and you can mouse over the sparklines to see the instantaneous rate, second by second.

Read More
Published by Baron Schwartz on Nov 12, 2014 2:23:00 AM

VividCortex Now Supports Amazon RDS For MySQL

We are excited to announce that we now support monitoring Amazon RDS for MySQL. Previously we relied exclusively on deep packet inspection, but since customers can’t install agents on RDS servers, we implemented support for the MySQL PERFORMANCE_SCHEMA tables in MySQL 5.6 and newer. Support for RDS performance monitoring is available immediately.

Read More
Published by Baron Schwartz on Nov 6, 2014 7:33:00 AM

Inside Distributed Architectures: RT In-Flight Requests And Blockers

VividCortex’s SaaS backend is a service-oriented architecture, which means that in addition to our external APIs that our agents and web UI use, we also have internal APIs. Deploy a bunch of API servers across a cluster of machines, hook everything up and make all the parts talk to each other, and what do you have? You have a distributed system that’s hard to troubleshoot, that’s what.

Read More
Published by Baron Schwartz on Nov 5, 2014 9:33:00 AM

Make Your Own Pumpkin Spice Latte

We all know that a programmer is a machine for turning caffeine into algorithms. Now that it’s autumn, and the marketers with their Big Data have figured out that we’ll buy nearly anything if it’s pumpkin spiced, it’s time for a comforting cup of pumpkin spice latte.

But unlike those served at big coffee chains, we’re going do make our own, because we are a DIY crowd, and we’re going to use actual pumpkin to do it. It’s easy and the results are delicious.

Read More
Published by Anna Navatsyk on Oct 30, 2014 10:12:00 AM

Win a Free Pass to Velocity Barcelona!

O’Reilly is quickly approaching and will be held Novemmber 17 - 19 in Barcelona. The folks at O’Reilly have offered us a free pass to give away, so we are offering this to you when you participate in the Trivia game. Answer any of the following questions and tweet your answers to @vividcortex with #velocitytrivia for your chance to win a 2-day pass. You can find the answers to the questions on the Velocity Conference website. Speed and accuracy is your friend in this game, as answers will not be accepted after November 6. The winner will be announced November 7.

The questions are below:

Q.1: If you want to build a culture and process that drives diversity in your recruiting, hiring and retaining people, what session should you attend?

Q.2: One of our keynote presentations features an open source project that helps front end develoers build faster, cheaper and more maintainable code using components. What is the presentation title and name of the project?

Q.3: Which Velocity Keynote session relates to this quote: “In times of drastic change, it is the learners who inherit the future.”?

Q.4: What are the “five reasons” to come to Velocity? Bolded reasons are fine. (hint, reading About Velocity will help).

Q.5: Which Velocity EU Event gives you 5 minutes to get on stage and talk your latest passion, idea or hack?

Q.6: Which session would you attend to learn more about polyfill performance and binary transmissions?

Happy guessing!

Read More
Published by Anna Navatsyk on Oct 29, 2014 6:20:00 AM

Free Webinar! Developing MySQL Applications with Go

Google’s Go language is increasingly popular for systems programming. We use Go extensively at VidivCortex and have come to appreciate its simplicity. It’s efficient and fun to program in Go, and it produces high-performance programs. Go is well suited for working with SQL databases, and has excellent MySQL drivers. Join us on Tuesday, November 11 at 2 PM EST (6 PM GMT) as Baron Schwartz, our CEO and Founder, discusses the following:

Read More
Published by Natasha Richter on Oct 27, 2014 8:50:00 AM

Debugging with Trace

When you are debugging a multi-threaded program, just adding Print() statements to your code does not always help. Coordinating several threads is not a straightforward task, and, often, the information you are printing may not be printed in the order you expect. This is especially a problem if your program is running multiple calls to the same function in different threads because there is not an inherently easy way to identify which thread printed which line. Trace solves this problem by tracking the threads.

Read More
Published by Preetam Jinka on Oct 16, 2014 3:03:00 AM

Smoke-Testing VividCortex's Agents With A Sanity Check Tool

One of our agents, vc-mysql-query, works by sniffing TCP traffic with libpcap and decoding the MySQL protocol. As you can imagine, it’s one of the most complicated portions of our codebase. It’s also difficult to test. We have a set of tests using tcpdump files of production MySQL traffic to deterministically test the sniffer code. We run the tcpdump files through the sniffer and check the generated output. The issue with this is that every time we add a new dump file, it’s a lot of manual work to know for sure what we should be seeing in the output. If the agent says that 100 SELECTs were run, how do we make sure it’s true?

Read More
Published by John Potocny on Oct 9, 2014 2:13:00 AM

Query Samples, Explained

The EXPLAIN command is one of MySQL’s most useful tools for understanding query performance. When you EXPLAIN a query, MySQL will return the plan created by the query optimizer. It also shows you how that query will be indexed and an estimate of how many rows are processed by that query. From this information, it is easy to see if your queries are taking advantage of table indexes or if you can change them for some extra performance. VividCortex provides a lot of information on query performance, including samples of the queries that are run against your database. Now, those samples will have EXPLAIN data for them too!

Read More
Published by Natasha Richter on Oct 6, 2014 7:28:00 AM

VividCortex and reCaptcha

When faced with adding a captcha to your forms, there are many libraries which can help you. This depends on your framework; if you are using Angular JS, you can use our captcha component to easily add one to your form.

Read More
Published by Anna Navatsyk on Oct 2, 2014 8:07:00 AM

Free eBook: Configuring MySQL for Performance Teaser

Have you donwloaded our latest eBook, Configuring MySQL for Performance? It offers simple advice on configuring the most important settings to maximize your MySQL performance.

Below are a couple tidbits:

The Default Storage Engine

The default_storage_engine setting is the engine used for tables that don’t specify an engine explicitly. This used to be MyISAM, but it’s changed to InnoDB more recently. This is a good thing; it’s the engine we’ve suggested as default for most use cases for years. Just check to ensure this is set as you desire. If you don’t know what engine to use, the right answer is probably InnoDB.

The Query Cache

The query cache (variables whose names begin query_cache ) should generally be disabled, unless you’re quite familiar with MySQL and its inner workings and know that it’ll help on your workload. You can find a lot of discussion online about the potential impacts of the query cache. Much of it misses the forest for the trees: the fact is that the query cache doesn’t scale well on modern multi-core hardware and can cause serious lockups of the whole server. It’s not worth dissecting the reasons in great detail here; if you want good detailed advice, see High Performance MySQL, 3rd Edition.

Interested in learinng more? Download the full edition here.

Read More
Published by Natasha Richter on Sep 30, 2014 10:31:00 AM

Opensource Update: Multitick

Multitick is an open source project which helps us coordinate worker goroutines. It broadcasts a time.Ticker to multiple receivers, all of which receive the same values, aligned to an offset. This makes the workers more testable by avoiding nondeterministic use of time.Tick() inside them.

Read More
Published by Anna Navatsyk on Sep 23, 2014 6:28:00 AM

Free Webinar! Knowing the Unknowable: Per-Query Metrics

At VividCortex, we use statistical techniques to infer the relationship between a statement’s execution and work that cannot be measured inside the server. For example, executing a multiple linear regression over a million samples of 500,000 independent variables is not feasible, and won’t produce good results even if it does run to completion. A much higher-performance algorithm is necessary. Join us on Tuesday, October 7 at 2 PM EDT (6 PM GMT) as Baron Schwartz, our CEO and Founder, discusses the following:

Read More
Published by Fernando Papa on Sep 22, 2014 6:59:00 AM

Using Netlink to Optimize Socket Statistics

This is a story of using low-level kernel interfaces to optimize an edge case one of our agents encountered in some servers. The TL;DR version is that accessing /proc/ can be very expensive if there are a lot of network connections, and the Netlink interface between userspace and kernel space is a much more efficient method.

Read More
Published by Anna Navatsyk on Sep 9, 2014 11:08:00 AM

VividCortex and Package pm

One of our open source projects is Package pm, which we’re using at VividCortex to inspect and manage API-sever programs. Package pm replaces an internal-only project, and we are happy to share it with the open source community.

Read More
Published by Anna Navatsyk on Aug 28, 2014 5:17:00 AM

Baron Schwartz to Speak on the Power of Regressions

Baron Schwartz, our CEO and founder, is speaking at the Performance and Capacity Conference by CMG on November 6th. CMG, the Computer Measurement Group is a not-for-profit organization that focuses on the efficiency and scalability of IT service delivery through measurement, quantitative analysis and forecasting. It has been a reputable, worldwide resource for best practices in the IT profession since 1974.

Read More
Published by Anna Navatsyk on Aug 27, 2014 6:25:00 AM

Upcoming Webinar: Log-Centered Infrastructure

For many companies, VividCortex included, managing crushing volume and velocity of data requires planning from the foundations of the infrastructure. A radically new approach is an IT structure in which the data log serves as the backbone.

Read More
Published by Anna Navatsyk on Aug 25, 2014 8:40:00 AM

VividCortex Presents at Velocity and Strata

VividCortex will be presenting several topics at upcoming Strata and Velocity conferences in New York and Barcelona. We’d love to meet you there. In addition to our talks, please catch us at author book signings and office hours in the O’Reilly booth in the expo hall. We’ll also have a booth at Velocity New York. Come say hi!

Read More
Published by Anna Navatsyk on Aug 14, 2014 5:20:00 AM

Please, Take a Seat - but Which One?

We recently researched furniture for our office transition and, in doing so, encountered a surprising number of seating options. Long gone are the days where every office chair is uniform; they have evolved to fit the needs of almost any seating preference. Here are the five we found most relevant to our search:

Read More
Published by Anna Navatsyk on Aug 12, 2014 2:00:00 AM

Product Evolution: Recent Features

We are constantly soliciting user feedback and drawing from our experience to make our tool as helpful as possible. Over the past few months, our brainiacs have rolled out a number of new features. Highlighted below are some of the major product enhancements:

Read More
Published by Baron Schwartz on Aug 10, 2014 2:48:00 AM

Report on August 1st Production Outage

On Friday August 1st, about 5pm EST, we had an outage in production that made our application and APIs partially unavailable. Though functionality was quickly restored, we lost data for a window of time. I want to personally apologize to our customers. We do take availability, correctness, and performance very seriously. We have made sure this problem won’t happen again.

Read More
Published by Baron Schwartz on Aug 7, 2014 9:15:00 AM

Drill Down With VividCortex's Events Dashboard

Over the last several months we’ve designed and built an events dashboard that lets you inspect very large amounts of system event data quickly (interactively). This feature was driven by customer requests and feedback. The initial proof of concept established its usefulness right away, when customers began remarking that they’d diagnosed server issues by noticing events such as database restarts, replication failures, and configuration changes. At least one customer told us this saved them a long wild-goose chase.

Read More
Published by Anna Navatsyk on Jul 31, 2014 6:45:00 AM

In Case You Missed It - Intro to ChatOps and Hubot

In this webinar, our brainiac Owen Zanzal speaks about the use of ChatOps at VividCortex.

He addresses:

  • Building Thing and the Hubot API
  • Deploying code with Thing
  • Teaching Hubot bookmarks
  • Teaching Hubot to fetch logs
  • Making Thing Funny

If you did not have a chance to join the webinar, we have linked to the slide deck, and this blog post contains extra detail. You can also register for a recording here.

Happy Chatting!

Read More
Published by Baron Schwartz on Jul 30, 2014 8:26:00 AM

VividCortex's New Agent Dashboard

We’re rolling out a brand-new feature by customer request: an agent management dashboard. You’ll find it in the left-hand navigation as usual. It shows all the agents running on all of your systems. You can sort, filter, and see CPU (user+system) and memory consumption. Here’s a screenshot of a random system on our production environment:

Read More
Published by Preetam Jinka on Jul 25, 2014 1:19:00 AM

Prepared Statement Samples

One of our most useful features is Top Queries, which allows users to examine various metrics for families of similar queries. Once a query family is selected, we can drill down into its query samples. Individual samples are shown on a scatterplot. Selecting individual samples will display its captured SQL, latency, errors, and more.

Read More
Published by Baron Schwartz on Jul 23, 2014 11:19:00 AM

View Per-Process Detail with VividCortex

You probably know about VividCortex’s Top Queries view, which shows MySQL query activity on one or many systems in industry-leading detail. But did you know you can also see what’s happening on the operating system? Not only do we capture fine-grained metrics about CPU, network, and so on, but we capture per-process metrics.

Read More
Published by Baron Schwartz on Jul 21, 2014 11:00:00 AM

We're Obsessed With Design

We obsess over design at VividCortex. In fact, I usually tell people that at least two-thirds of our product’s value is in the design and user interface. A great product has to be well-designed — it has to be productive and pleasant to use. This is not optional. It’s make-or-break.

Read More
Published by Baron Schwartz on Jul 21, 2014 6:09:00 AM

MySQL Type Conversion Rules

When you send a query to MySQL, you usually don’t need to think explicitly about the types of the expressions in your query. If you compare this to a lot of programming languages, you’ll find that it’s not always the way things work. In strongly typed languages like Java, for example, typing is very strict.

Read More
Published by Baron Schwartz on Jul 15, 2014 8:23:00 AM

Introducing VividCortex's New Time Selector

We’ve recently deployed a new time selector based on extensive customer feedback. It’s designed and built with love by our talented team. Our previous time selector offered only a handful of preset choices, but the new time selector allows you to smoothly slide along a variable scale as well as clicking on presets.

Read More
Published by Anna Navatsyk on Jul 14, 2014 5:40:00 AM

The Importance of Being Happy

Finding fulfillment at your job is important, especially on Mondays. A stimulating environment inspires people to do their best work. At VividCortex, we want everyone to look forward to getting out of bed each morning. We want to ensure we are doing everything within our power to unblock and empower employees. One way we gauge our effectiveness is through an employee happiness survey - a pulse check, if you will.

Read More
Published by Baron Schwartz on Jul 2, 2014 3:23:00 AM

See Per-Database Activity With VividCortex

Is your application multi-tenancy, where each tenant lives in its own database? We have such a scenario at VividCortex, and many of our customers do as well. Wouldn’t it be nice to understand which tenants are causing more load than others, which are the noisy neighbors impacting others’ experience, and so on? Well, now you can!

Read More
Published by Baron Schwartz on Jul 1, 2014 4:11:00 AM

Composition Instead of Inheritance in Go

Looking back, I find object-oriented programming in languages like Java and C++ to be incredibly confusing. There are so many special rules and things to know. By contrast, Go gives me tools for everything I want to do, and it’s so simple and explicit.

Read More
Published by Anna Navatsyk on Jun 26, 2014 9:27:00 AM

The Hidden Cost of Data Operations eBook

A few months ago, we asked DBAs, SysAdmins and others who manage servers to fill out a survey assessing their resources, workload and tools. We received 150+ responses from those at mid-size to large web scale operations companies that work with systems built around open source technologies. We have compiled the data, and the telling results are available in this eBook.

Read More
Published by Baron Schwartz on Jun 23, 2014 9:06:00 AM

Discovering Query Bugs by TCP Inspection

The MySQL wire protocol includes useful result data and metadata, such as warnings and errors raised by the query. These often indicate bugs that will cause problems you may not discover for a long time, when the cleanup can be costly and damaging.

Read More
Published by Anna Navatsyk on Jun 11, 2014 8:37:00 AM

VividCortex Predicts the World Cup

Today marks the kick off of a sporting spectacle that rallies nations. In honor of VividCortex’s first World Cup, below are a few employee predictions indicating clear biases and varying levels of knowledge.

Read More
Published by Owen Zanzal on Jun 2, 2014 7:57:00 AM

ChatOps At VividCortex

We are huge believers of ChatOps at VividCortex. If you are unfamiliar with the term “ChatOps”, it was coined by GitHub, and the core idea is simple: make common commands executable from within your company’s chatroom via a chatbot. (GitHub opensourced their chatbot Hubot.) This allows everyone in that room to see you execute those commands, making the commands immediately accessible. New hires learn commands by seeing current employees execute them regularly. These simple concepts turn out to be very powerful. GitHub’s implementation of ChatOps is amazing and has inspired me to spearhead ChatOpts within our office.

Read More
Published by Baron Schwartz on May 6, 2014 6:10:00 AM

Handle Errors or Pass the Buck: Pick One

In Go, errors are types like any other, and are not “exceptional” in any way. The consequences are subtle and very beneficial. Newcomers to the language are usually struck by how verbose the error handling tends to be:

Read More
Published by Baron Schwartz on May 1, 2014 9:08:00 AM

Updates from VividCortex

What’s new in VividCortex? We’ll do these blog posts once a week to let you know all of the changes such as new features, bug fixes, performance improvements, and even a peek into the future.

Read More
Published by Baron Schwartz on Apr 24, 2014 11:00:00 AM

Adaptive Fault Detection Versus Anomaly Detection

We get a lot of questions about what our Adaptive Fault Detection algorithm really is and how it works. Although I can’t tell you exact details, I want to explain a few things about what our algorithm is and what kind of results it can deliver. I get a lot of questions about it that assume it’s just anomaly detection, so I’ll cover that. I will also show a couple of representative faults our system has detected.

Read More
Published by Kyle Redinger on Apr 24, 2014 8:13:00 AM

Join Our Team! We're Hiring!

We’re Hiring!

VividCortex has several open positions, both technical and non-technical. All of these positions are fulltime, based in Charlottesville, VA or Montevideo, Uruguay. Our benefits include:

  • Fulltime salaried positions
  • Working with some of the coolest customers in the world
  • Flexible work environment, unlimited vacation and remote working opportunities
  • Competitive salaries
  • Option grants
  • Change to work on challenge startup challenges
  • Use technologies like Go, Angular, D3, MySQL, Kafka and more bleeding edge stuff
  • Full healthcare and dental coverage for US employees
  • Lunches, cellphone and a macbook

Charlottesville Office Manager & Startup Ninja

Ideally, the candidate is incredibly organized, knows how to get stuff done, a strong communicator, and wants to be part of a fast-growing startup. The candidate must possess immaculate writing skills and be self-directed when it comes to learning something new. The candidate should also have a passion for technology and enjoy working in an engineering culture.

Systems Programming

Our agents, backend systems, and infrastructure are all written in Go (it’s awesome). We are ramping up to handle a jaw-dropping amount of incoming data. Data storage involves MySQL and other technologies in a fault-tolerant, distributed environment.

Operations and System Administration

We’re hosted in Amazon AWS and looking for people who do developer and sysops types of work — writing some server software, setting up some machines, whipping up some Ansible code, and so on.

Front End Development

The web app is backed by a small amount of PHP and Symfony2, and fronted by Angular and D3.js, CSS (Less), and HTML5. There’s a lot of API interaction from JavaScript, and a complex single-page user interface with rich data visualization that’s key for making our customers rave about the experience, not just the technology. We need people who can work in Angular, D3, design, styling, and so forth.

**If you are interested in any of these positions, send a short intro and resume/github/linkedin to jobs AT ourdomain.com. **

Read More
Published by Baron Schwartz on Apr 22, 2014 7:25:00 AM

Visualizing The Impact Of An Index Change

One of our top queries updates a table called host_agent with status information from agents checking in with the APIs periodically. I was always a little surprised at how “hot” this particular query is. If you’d asked my intuition, I wouldn’t have thought it would be in the top 25.

Read More
Published by Baron Schwartz on Apr 21, 2014 10:16:00 AM

Monitoring is Dead, Long Live Performance Management

At VividCortex, we are explicitly not a monitoring company. We are addressing the human-to-data gap by building tools that help people manage thousands of servers effectively. This tension is caused by trying to run increasingly large numbers of servers with fixed staff and budget, using relatively immature software that doesn’t have sophisticated tools. It’s a worst-case scenario for the operations staff: more data, more servers, worse tools.

Read More
Published by Baron Schwartz on Apr 14, 2014 2:21:00 AM

Examining Query Samples with VividCortex

Customers love our Top Queries feature, which lets them rank queries by a metric such as overall execution time or count. This is a great way to examine entire families of similar queries together. We group queries by digesting out the literals, normalizing whitespace, and so forth.

Read More
Published by Baron Schwartz on Apr 7, 2014 5:54:00 AM

How Accurate Are Computed Columns?

If you’ve used our Top Queries feature to profile your MySQL server’s queries, perhaps you noticed the little icon to choose what columns display in the table. Click that icon and you can add columns for computing per-query CPU, IO, and other resources. Here’s a screenshot:

Read More
Published by Kyle Redinger on Apr 3, 2014 8:20:00 AM

VividCortex Launches at DEMO Enterprise

SAN FRANCISCO, CA–(Marketwired - Apr 3, 2014) - VividCortex today announced the launch of its software-as-a-service database performance management solution at DEMO Enterprise 2014 in San Francisco, renowned for launching the most innovative companies in enterprise, mobile, cloud computing, consumer, and social media technologies.

Read More
Published by Kyle Redinger on Mar 26, 2014 8:13:00 AM

Convertible Note Fundraising Documents

Fundraising can present a formidable challenge for new entrepreneurs in early stage startups. To simplify early stage fundraising, accept the market for money. Terms shouldn’t limit an investor’s interest in you; the focus should be on you and the idea.

Read More
Published by Baron Schwartz on Mar 24, 2014 2:25:00 AM

Query Time Is Not Server Load

For many years my writing, teaching, coaching, performance optimization, and consulting has assumed the following to be true: to reduce load on a MySQL server and free up resources, sort queries by total time and work from the top. The problem is that is wrong. In this post I’ll explain how how we discovered this and why it matters.

Read More
Published by Baron Schwartz on Mar 17, 2014 10:26:00 AM

Multi-Threaded Replication in MySQL

For years, people lamented that MySQL’s replication was single-threaded. Peter, Vadim, and I griped about this in High Performance MySQL for two whole editions, as well as a bunch of blog posts.

Read More
Published by Kyle Redinger on Mar 17, 2014 8:18:00 AM

VividCortex Will Launch at DEMO Enterprise on April 3rd

We are very excited to announce that we will be launching VividCortex at DEMO Enterprise on April 3rd, 2014 in San Francisco, CA. We’re really excited to showcase our product to the world, in a live event, in front of thousands of journalists, technologists, and anyone else who may be listening.

About Demo

DEMO is a prestigious event where some of the coolest products in the world have launched. VMWare, Evernote, Salesforce, and many others have launched there. We expect it to generate some great press about our company and we’re honored to be part of it.

Big Thanks

Most importantly, we wanted to thank all of our beta users for helping validate, direct, and participate in our mission. Baron and I founded the company about 18 months ago and the mission continues to get more exciting.

We’ve raised $2.7 million in funding, we’ve added team members, we have some of the most impressive companies in the world benefitting from our service. Our product has morphed into something that’s ready for the world to see.

We can’t name companies, but I also wanted to give a big thanks to our early customers. Without you, the business wouldn’t be where it is today. Your purchase is an investment in our future and and investment in making your life much better.

Whether you were an active beta user, a customer, an advisor, investor or just a fly on the wall, thank you for believing in us. We’re going to do great things.

We’re looking to expand our business later this year, so stay tuned for news on future fundings, new services and lots more great features.

##About the Launch

Our embargo ends on April 2nd at 9:01PM Pacific time, and we’ll be saying a lot more about the product at that time. We plan to give users free access to a fully-featured trial version of our product, so you can give us feedback, see if it’s a good fit for your organization, and try things out.

If you want to be notified as soon as everything is available, please signup for our beta. We’ll email you as soon as we are out of beta.

If you want a sneak peak, and an interview, and are a member of the press that honors embargoes, we’ll be happy to chat with you.

##Technical Product Reveal Webinar

If you are interested in seeing the product demoed and/or have questions, register to attend our webinar on April 8th at 1PM ET.

Read More
Published by Kyle Redinger on Mar 5, 2014 5:48:00 AM

Hiring Inside Sales

Join us as an Inside Sales team member

Hello awesome hustlers, sales people and lovers of tech. We’re hiring an inside salesperson to build relationships and close deals with our customers.

This position reports directly to me, and has the opportunity to grow into various roles depending on the candidates strengths and the company’s needs.

You would be working for a venture-funded startup, at a pivotal time, during our launch and scale period. This would be a great opportunity to get in on the ground floor of a startup. We have an impressive list of clients, investors and advisors.

Ideally, you have a deep exposure to technology, can talk with people in technical roles, enjoy setting phone/virtual meetings with a minimum of 100 leads per month, can manage salesforce, favor performance-based pay, and possess a friendly tenacity to get stuff through the buying process. Sales experience is a must.

This position is full-time, based in Charlottesville, with unlimited vacation, an awesome healthcare and dental insurance plan, and in a high-paced, startup environment. The pay is heavily performance based, and can include equity.

If you are interested, send an email to jobs@vividcortex.com with

  • your resume
  • a paragraph selling yourself
  • a list of relevant experiences

Thanks!

Read More
Published by Kyle Redinger on Mar 4, 2014 4:55:00 AM

State of Big Data Markets

I recently had the pleasure of presenting the “State of Big Data Markets” to a non-technical audience. Big data might be the biggest hyped term of 20132014, but the interesting part for VividCortex and many others in the industry might not be as obvious.

Big data disrupts traditional technology vendors and creates an emerging world order where legacy technologies and business models fall short. Sure, like any industry, legacy players will become entrenched rent collectors, but the innovators are a new breed of software and services, built on top of increasingly commoditized hardware, with costs that are miniature compared to just months ago.

Read More
Published by Baron Schwartz on Feb 25, 2014 11:24:00 AM

Performance Schema, Slow Query Log, and TCP Decoding

Peter Zaitsev posted Percona’s answer to the question, “why use the slow query log instead of the PERFORMANCE_SCHEMA tables to gather information about query execution and timing?” At VividCortex, we don’t currently use the PERFORMANCE_SCHEMA for this either. Instead, we capture and time queries by sniffing the TCP traffic to the server, and decoding the MySQL protocol. We extract rich information from the protocol, including the query text, timings, errors, warnings, query flags, and more.

In this post I’ll explain why we do this and how it works relative to other options.

Query Capture Methods

What options exist for measuring queries the server executes? Here’s a short list.

  • query logs
  • PERFORMANCE_SCHEMA
  • packet inspection
  • proxies
  • tools that poll commands such as SHOW PROCESSLIST
  • a plugin for the audit log
  • application-side instrumentation in drivers

Most of these are unsuitable for use in real production applications.

For example, polling the SHOW PROCESSLIST command misses most queries, gives very poor timing resolution, and doesn’t provide much information about query execution. Worse, it adds potentially large amounts of overhead and locking to the server. It is not an option a serious MySQL user should consider. I am biased, but I also believe you should stay away from tools that use this technique.

An audit plugin is not great, because the audit plugin API doesn’t give access to data you’d probably like to have. (I know of audit plugins that meddle with server internals at the level of rewriting bits of the server’s memory, for example. Risky.)

Proxies are right out for most people I know; it requires rewiring the app, and becomes a single point of failure. Your application should never be down because your monitoring tool crashed.

Application-side instrumentation such as the connection libraries is more reasonable, but gives an application-centric view of the server’s workload, which is just a subset of what really goes on.

For a bit of context, Oracle’s MySQL Enterprise Monitor (MEM) used to capture queries either via MySQL Proxy or via the connectors such as Connector/J. I have never known anyone to use MEM with MySQL Proxy in production; they sometimes enabled it in development environments, but that was pretty much a deal-killer for production deployments. Clearly Oracle has built the PERFORMANCE_SCHEMA so they can get away from the limitations of proxies and connector-driven captures.

In the end the best options really are logs, PERFORMANCE_SCHEMA, and deep packet inspection on the network.

Why We Use Packet Inspection

Why do we capture network traffic and decode the conversations between clients and servers?

To begin with, I will echo Peter’s points about compatibility. The reality is that although PERFORMANCE_SCHEMA is nice, it’s not available in most server installations, and our product needs to work on older versions of the server. Until PS is widely deployed, it’s not good enough for us or our customers.

Compatibility is part of the reason we don’t use logs. As you can read in Peter’s post, Percona’s tools rely on a patched/forked/enhanced version of the server, depending on what terminology you’d like to use. The patches add valuable information to the query log, but there are also several crucial enhancements to the logging functionality that help mitigate the performance overhead of logging. Without these, using the slow query log becomes impractical and potentially even dangerous. Although Percona Server and MariaDB are great products, the vast majority of users will prefer Oracle’s official version of the MySQL server. That’s why, for maximum compatibility with many server versions, we cannot use the slow query log.

On the other hand, if we were to use the query logs on Percona Server or MariaDB, we would face a different problem. The features that make query logging safer and more feasible do so in part by not logging a lot of queries. We want to observe as much server activity as possible, not just 1% or so of the queries.

As an aside (a topic for a different blog post), the slow query log and PERFORMANCE_SCHEMA provide information we’re not interested in, and don’t provide some of the data we want. We get the data we want in less intrusive, more accurate ways. I will speak in detail about this at Percona Live in April (and share source code so you can reproduce our work). The detail we can extract about queries is amazing, and the overhead is very low. In other words, although it might seem like you can’t get all the useful information from the protocol, and it’s only available internally to the server, that turns out to be not true in the general case. Stay tuned for more on this.

Performance Concerns

For performance reasons, TCP packet inspection and decoding is much better than log parsing. I created and worked for years on Percona Toolkit, including the slow-query-log parser in pt-query-digest, the tool that Percona uses to extract query metrics from logs. I know a great deal about log parsing and its pros and cons. When I founded VividCortex, I could have written new and more efficient tools to parse slow query logs, but I had already decided that it was not the way to go:

  1. It adds high overhead on the server. We do not want to intrude upon the systems we observe, and by Peter’s admission logging all queries to the slow query log can easily approach 40% overhead in synthetic benchmarks. I would add that this is not unrealistic in real applications. The overhead is both internal to the server (locking during log writes, for example) and external (increased I/O, CPU, and memory consumption).
  2. Log files aren’t just expensive to write, they’re expensive to read and parse. I spent a huge amount of time over the years trying to optimize pt-query-digest’s log parsing routines, but never got it to levels of CPU and memory consumption that satisfied me. Ask anyone who does this a lot and they’ll agree: you can log 5 minutes worth of queries on a busy server and pt-query-digest will take gigabytes of memory and run at 100% CPU utilization for a long time trying to parse it. That means that even if there’s lots of spare memory and CPU for it to use, it will not be able to keep up with the log on a busy server! Part of this is due to the practically unparsable log format, which has not been improved by all the patching that’s been added to it. Part of it is also inherent to the way pt-query-digest does things (a lot of regular expressions), which could be improved by using other methods.

Again, some of these problems can be addressed by doing things like sampling, and that is what most people who use pt-query-digest in practice do. For example, I was visiting some friends at a large enterprise company last week. They enable query logs for a few minutes once an hour and log only 1% of the queries in the server during that time. This is far from a complete view of what’s running in your systems.

Peter mentions several other features added to Percona Server to attempt to tweak the slow query logging functionality to make it a more acceptable tool for measuring what the server does, but in the end it’s clear that this is not a solution for the majority of users. I’m not saying it’s not good for the way Percona’s customers operate, but we are building a solution that offers first-class functionality for everyone, not just people who use a specific fork and are content to measure a fraction of a percent of the queries on their systems.

How much overhead do our agents cause on customer systems? Since we track information on the operating system statistics (including per-process statistics such as CPU and memory), we know exactly how much overhead we add to systems where our agents run. Over the last 30 minutes as I write this, our highest-traffic server is seeing the query agent consume 46MB of memory and 4.07% of a single CPU. That’s with 100% traffic capture — millions of queries being captured, decoded, digested, and measured. Some of our customers have busier servers than ours, there are worst-case scenarios that can cause more CPU usage, and we have seen our agents use more CPU than we want, but in general the overhead is very low. We are also continuing to lower the amount of overhead and put safeguards in place against the edge cases.

Configuration, Setup, and Disruptiveness

There’s another important point to mention about analyzing log files: the setup for it is non-trivial. There are a lot of moving pieces to put into place, even in a service that provides as much canned functionality as possible for you. If you’re setting up and using something like Box Anemometer, a lot more work is necessary. These are not “bad” things per se, but we want to lower the barriers to using our service as much as possible, so any configuration or setup is something we want to avoid.

Beyond ease of adoption, there are risks to consider too. In addition to performance overhead, query logging can cause problems if things aren’t done correctly. When I was at Percona I studied the causes of production downtime, and wrote two white papers on the results. Full disks were the leading cause of downtime in the leading category of causes. I have seen servers get into trouble many times because the slow query log filled the disk up. Tools to avoid this, such as logrotate, can cause their own problems in many cases. There are just too many ways it can go wrong in my experience.

By contrast, VividCortex’s setup is as simple as curl-ing an install script and giving it your API token. Everything else is automatic, and our agents will discover everything in the environment and self-configure. It’s important to note that they don’t interfere or alter anything such as server configuration. No need to configure the server, no delicate dances with enabling and disabling the logs periodically, no rotating and emptying the logs, and so on. Packet inspection lets us avoid all of this and just eavesdrop on the system without interacting with it.

This works on 100 servers as easily as it works on 1; it’s so easy it’s practically magical, and takes less than 15 seconds. We’ve worked hard to make our installation process very smooth. If we had opted instead to consume query logs, we wouldn’t have been able to make things so simple and unintrusive.

Flexibility

Packet inspection gives us a lot of flexibility and represents a smooth upgrade path to measuring more things in the future. For example, packet capture can be done on the database server as we currently do; we could use port mirroring to capture traffic; or we could capture traffic from the application servers. All of these offer advantages and make it possible to use our service in more ways and in a greater variety of environments.

Log analysis, by contrast, is much less flexible. And PERFORMANCE_SCHEMA is also constrained in some ways as well (i.e. it cannot be done passively on a remote system, since it is a part of the server internals.)

Shortcomings of Packet Inspection

That’s all great, but what are the drawbacks? I will be very frank and say that there are a few.

  1. We can only observe queries sent over the network. We don’t see queries sent through Unix sockets or via replication. We also can’t see queries run from stored procedures. We understand the binary protocol and the compressed protocol, and we support prepared statements, but if a statement is prepared before our agent starts observing the traffic, we won’t know what statement was executed. In my experience these are not a big deal, and our customers thus far agree. Note that we will certainly support the PERFORMANCE_SCHEMA in the future, so these limitations aren’t permanent.
  2. The agents need to run as root. Our customers tell us this isn’t a big deal, since they already run Chef/Puppet as root, and other services are similarly run with root privileges. Even Percona’s blog post about installing their agents mentions that they should run as root. So far, no one has even raised this as a concern.

In the end, deep packet inspection of the type we do is a compelling way to capture and measure queries to the server. It can be done completely passively without interfering with the server. With excellent engineering it can be made trivially low-overhead. It captures all the information about the query that we need. It doesn’t rely on any specific server features or configuration.

The main downside, really, is that it’s a hard engineering problem that takes a lot of careful programming to get right and to make it efficient. I think it’s better than all the other options including the PERFORMANCE_SCHEMA (although I am sure some will disagree with me on that). I think we’ll always prefer packet inspection, even after we add support for the PERFORMANCE_SCHEMA.

If you’re interested in becoming a beta customer, please sign up on our waitlist and say a few words about your use case. We are currently in a very limited private beta, and are occasionally inviting a few more users to help us test the product.

Read More
Published by Kyle Redinger on Jan 21, 2014 6:01:00 AM

Top 10 things you need to do to get an internship

We’ve interviewed a lot of students, mostly from UVa, who are interested in working at VividCortex. All of them are bright and capable, but we’ve had some very unprofessional experiences with students. The intention of this post is not to talk about the technical shortcomings (we have some advice in a future post) or critique, but rather, basic ways to hack your way into a cool internship. This advice comes directly from actual experiences interviewing 20+ students in the last few months. It’s generally applicable to all job seekers, but targeted to students.

Read More
Published by Baron Schwartz on Jan 16, 2014 8:32:00 AM

Announcing a Shiny New Johnny Deps

We’ve released a shining^Wshiny new version of Johnny Deps, our dependency manager for Go. The new version is simpler, cleaner, and more powerful. It also has a shorter name, jd, instead of being called johnny_deps. It’s not backwards compatible, but it’s pretty similar.

Read More
Published by Kyle Redinger on Dec 19, 2013 9:57:00 AM

Why Tom Tom

The Tom Tom Founders Festival celebrates the arts, innovation, entrepreneurship and ideas. This year it’s taking place April 10th through April 14th and we wanted to invite all our clients, investors, friends and supporters to come hang out with us in Charlottesville.

Read More
Published by Baron Schwartz on Dec 9, 2013 4:20:00 AM

Analysis of PayPal's Node-vs-Java benchmarks

I was interested in the data shown in PayPal’s blog post on switching from Java to Node.js, for two reasons. First, at a glance both sets of benchmark data looked very clean and fittable to the Universal Scalability Law to assess where the bottlenecks lie. Second, stepping back and taking a broader view of the blog post and the supporting benchmarks shows immediately that either something is very wrong, or we’re not hearing the whole story.

Read More
Published by Baron Schwartz on Nov 26, 2013 9:21:00 AM

Easy map and slice declarations in Go

In Go there are a couple of variable declaration syntaxes. Go uses type inference, so you don’t have to repeat yourself. Maps (hash tables; dictionaries), and slices (views onto arrays) always felt clumsy to me, because they use the make() function. I’m also a fan of using the := declaration-and-assignment syntax most of the time, as opposed to explicit variable declarations:

Read More
Published by Baron Schwartz on Nov 3, 2013 6:16:00 AM

How Much Data Does VividCortex Collect?

I often hear companies (or their vendors) talking about how much data they deal with. The numbers are frequently quoted per day, or sometimes even per month. The thing is, they’re often not very impressive. We process 25 million events per day sounds like a lot until you divide by 86400 (the number of seconds in a day) and realize that isn’t even 300 per second.

Read More
Published by Baron Schwartz on Oct 30, 2013 4:58:00 AM

StrataConf Interview with Baron and Kyle

Kyle and I were interviewed at O’Reilly Strata Conference and Hadoop World yesterday. The video is linked below. The themes of the interview questions came from our talks at Strata, both of which were about making large data digestible by humans and deferring to humans what they’re better at. (See Kyle’s Ignite talk for more on that.)

Read More
Published by Baron Schwartz on Oct 24, 2013 4:08:00 AM

Slides from Devops Days NYC

I presented last Friday at Devops Days NYC. After my rather cerebral “envision the future” talk at Velocity earlier in the week, I tried to come back to earth a little and talk about how to improve the monitoring tools we use right now — making the best of what we have. I think the title conveys the gist of it: monitor some of the things. As someone pointed out on Twitter afterwards, though, I was not being careful enough with my language. I was mostly focusing on alerts, not monitoring-in-general.

Read More
Published by Kyle Redinger on Oct 2, 2013 9:41:00 AM

Making You Happy: Live Customer Stories

We just soft-launched to a handful of early customers yesterday and things are looking pretty good. We’re starting to achieve our goals of making our users very happy. It’s not marketing talk. One of our customers just mentioned the following to us:

Read More
Published by Baron Schwartz on Sep 10, 2013 2:26:00 AM

100% Reproducible Builds in Go

This post was featured in our July 2015 anthology of most popular VividCortex blog posts of all time. To see more of our top content, check out that list here

One of the nice things about Go is statically compiled binaries, which don’t have external dependencies. There are lots of reasons this is a great thing. On the other hand, as we’ve

Read More
Published by Kyle Redinger on Sep 4, 2013 9:10:00 AM

VividCortex User Experience Goals for a Product Launch

Is everything UX?” is a rhetorical title to a post about the design behind our product. Baron and I agree that a great UI is most of the value we provide to our customers. We emphasize the thought process that goes into good design thinking. When you involve design and a customer-centric approach in the product creation process, it makes the product much better.

Read More
Published by Kyle Redinger on Aug 7, 2013 8:34:00 AM

Database-Performance Pioneer VividCortex Raises $2 Million

VividCortex, a pioneer in the emerging field of database performance management, announced it has raised $2 million in seed-stage funding from investors including Jaffray Woodriff, who runs the Quantitative Investment Management fund, Battery Ventures and various angels. The company—co-founded last year by leading MySQL authority Baron Schwartz—plans to use the funds for product development and additional hiring.

VividCortex’s technology helps Web-scale operations monitor, diagnose and fix performance problems that affect end users, including slow website performance. The software-as-a-service product can help large and small enterprises replace more-expensive, internal monitoring equipment and make database administrators more efficient.

The company’s product is now in beta and being used by customers in fields such as e-commerce, media and finance.

“We are delighted to work with Jaffray and Battery. We are investing significantly in engineering and infrastructure, and their support and expertise is exactly what we need to keep hiring and building, as well as to mature our sales and marketing models,” said VividCortex Chief Executive Schwartz, who previously led sales, marketing, consulting and software engineering at Percona, a prominent MySQL consulting and support company.

“Existing application-performance monitoring products are good at telling you that your database has a performance problem, but most of them come up short in terms of diagnostics and actual remedies,” Schwartz added. “We feel we are filling a critical market niche as ‘big data’ becomes a reality for more and more Web businesses.”

Schwartz started VividCortex with co-founder Kyle Redinger last year after seeing first-hand, as a consultant, many consumer-Internet companies struggling with Web-performance issues as the volumes of data they processed grew exponentially. Schwartz and Redinger set out to offer highly intelligent tools to manage complex systems at the server and database level, instead of just for specific applications and networks.

“Over the past five years SaaS monitoring tools have revolutionized the Web and application tiers of the enterprise stack, but the database has remained a black box,” said Alex Benik, a principal at Battery. “I’m excited to work with Baron and the VividCortex team as they address what is, more often than not, the root cause of many application-performance issues.”

Schwartz is the author of “High Performance MySQL: Optimization, Backups and Replication”, a well-known book about MySQL database management published by O’Reilly Media.

Read More
Published by Kyle Redinger on Jul 31, 2013 4:04:00 AM

Six Open-Source Projects in Six Weeks!

We are avid supporters of open-source at VividCortex and of our team when they contribute to the open-source world. It’s a win for us and a win for the community. While we are an early stage startup, we’re happy to make this one of our priorities. We wanted to thank all our team members for their awesome contribution and for setting a big record at VividCortex: six open-source projects in six weeks!

Read More
Published by Baron Schwartz on Jul 30, 2013 10:14:00 AM

Writing resilient programs with Go and robustly.Run()

Google’s Go language makes it easy to do things that are hard in many other languages. Making highly resilient programs is one of them, but it doesn’t seem to be discussed a lot. Crashes will happen eventually in every program, and it’s good to know how the program will behave even when you have no idea how or why it will crash.

Read More
Published by Baron Schwartz on Jul 23, 2013 3:42:00 AM

A fast Go library for exponential moving averages

We’ve open-sourced our Go ewma library. It computes fast exponentially weighted moving averages over streams of numbers. There are two implementations with varying properties, and the library encourages the use of a MovingAverager interface for ease of building and using more implementations.

Read More
Published by Kyle Redinger on Jul 19, 2013 9:29:00 AM

Low-Tech Solution to Project Management in Remote Working Environments

We’ve been struggling with project management tools. We are extremely GitHub focused (i.e. every development task is a github issue), and no project management tools let us automatically combine the granularity of GitHub issues with a project-level view. In short, it creates redundant work flow where we need to manage two different tools. Huboard and Strikebase are working on solutions to this problem, so in the meantime we’ve hacked together a relatively low-tech solution.

Read More