Gametime United is the market's most innovative solution for users who want to purchase last minute tickets for live events in their city. As a mobile platform, Gametime offers "curated lists" of available ticket options from hundreds of events, examining over 10,000 potential seats in the process.
Gametime's technical operations are handled by their internal engineering team, a group that spans a wide range of roles, including all of software development and quality assurance. Every member of the team is a full stack engineer; there is no dedicated DBA. As Charles Rice, Gametime's Vice President of Engineering, told us, the team is further broken down into three client teams — Android, iOS, and Mobile Web — and two backend teams, which handle selling and supply. Gametime's databases are the only standalone servers they use, and they run MongoDB 3.2.
"Without any doubt, VividCortex continues to pay for itself... Checking VividCortex is now part of my daily routine — it surfaces items that are truly actionable. I can look at a dashboard and say, 'I should review that particular query,' or, 'I should look at that particular function.' In fact, we changed our MongoDB cluster as a result of those insights."
—Charles Rice, VP of Engineering
Before VividCortex, Flying Blind Without a Solution
Prior to installing VividCortex, Gametime's team had been seeing behavior in their database that they couldn't completely understand. For instance, Charles explained, the system's CPU utilization was hard to parse, and the number of transactions they registered versus their number of databases didn’t line up with their amount of caching. They saw occasional spikes on read replicas that made performance inconsistent. They began to wonder whether their systems were sized correctly; they asked if their hardware was insufficient. Faced with layers of uncertainty, "We started to really feel blind," Charles said.
Gametime had been using Mongo's own out-of-the-box monitoring tools, which they could use to view slow query logs and other basic metrics, but, Charles said, these insights weren't in a presentable format and were difficult to utilize and optimize. Mongo's open source tool set didn't seem strong enough, especially when the team wanted to really understand what was happening in their database. "We had done some monitoring," Charles said, "but really only high-level CPU utilization, disk utilization, the number of queries — nothing really beyond that, nothing really actionable."
However, the team soon found VividCortex and discovered that it was designed to do exactly what they was looking for: deep, specialized, interactive database monitoring for MongoDB. "VividCortex certainly democratizes who can view database problems," Charles said. "I can now have junior engineers or interns take a look, and even they can locate something like a slow query."
Gametime's team has found three key benefits in VividCortex: They've seen it enable cost reduction; it's improved server reliability, by helping the team identify hidden, critical problems — what Charles called " black swans"; and VividCortex has become part of the team's ongoing development process, ensuring that bad queries don't creep into the database. "A bad query shows up very quickly in VividCortex's profiler," Charles said.
A specific use case that Gametime shared with us involved an instance when VividCortex helped reveal how a replication error was causing serious slowdowns in Gametime's system. They had originally been using one write master and six read replicas; whenever they suffered a performance issue, the team would respond by adding a new read replica. It wasn't until they used VividCortex that they saw that this was "exactly the wrong thing to do." During the replication process, the ops log for Mongo is very heavy weight, partially because it's a NoSQL database. Each read replica was actually hurting Gametime's performance — they were able to make the database cluster faster by shutting down servers, a revelation only discovered through VividCortex.
Now, Charles uses VividCortex on a daily basis, as part of his routine, to help ensure the company's MongoDB cluster is running at peak performance. His whole team benefits from VividCortex's democratization of the database, making discoveries possible for everyone, ops and developers alike—even without a specialized DBA onboard.
"VividCortex was an eye opener. Our ops process had been, 'Oh, we have slow performance, we’re getting too many reads, let’s add another read replica. Surely that’s the right answer.' But no, as VividCortex showed us, that wasn't the right answer."