How Much Data Does VividCortex Collect?

Posted by Baron Schwartz on Nov 3, 2013 6:16:00 AM

I often hear companies (or their vendors) talking about how much data they deal with. The numbers are frequently quoted per day, or sometimes even per month. The thing is, they’re often not very impressive. We process 25 million events per day sounds like a lot until you divide by 86400 (the number of seconds in a day) and realize that isn’t even 300 per second.


It’s more like a garden hose than a firehose, really.

That’s why I prefer to deal with consistently dimensioned events: everything should be per-second for purposes of computer processing. If you’re talking to the CFO about sales figures, of course another unit is appropriate. But rates of computer processing should almost always be per-second.

So how much data does VividCortex process? Well, I didn’t know the answer to that, because it’s not really that interesting to me. But just for grins, I checked how many metrics we ingested two days ago, and it was nearly 3.2 billion metrics per day nearly 37,000 metrics per second.

Which is, in my view, entirely unimpressive. Because we’re barely getting started. 37k metrics/second is nothing, peanuts, trifling. When we hit 40 million metrics per second (we’ve projected that to happen within our horizon), I’ll be more interested. If you’re doing the math, that’s 3.45 trillion per day. More importantly, we try to gather useful, meaningful metrics. We don’t just scrape everything we can off of systems.

When we demo our tools, people often say “how do you make it respond so fast?” Generally speaking, our platform ingests and crunches data very fast. (Not always — we don’t always have instantaneous responses. We’re still early and have left a lot of performance optimization for later, after we make sure that we’re building the right thing.)

I think it’s more interesting to see how small and low-performance our platform’s infrastructure is; we’re very efficient and we’re not even using high-end hardware. I discussed some of our philosophy on this topic last week at Strata. We’ll deploy “big iron” servers later, but for now we’re able to make our systems work effectively with very limited resources.

If you’d like to join our team, we’re hiring. We’d love to talk to you.

Recent Posts

Posts by Topic

see all