March 16, 2016

Sampling a Stream of Events With a Probabilistic Sketch


Stream processing is a hot topic today. As modern Big Data processing systems have evolved, stream processing has become recognized as a first-class citizen in the toolbox. That’s because when you take away the how of Big Data and look at the underlying goals and end results, deriving real-time insights from huge, high-velocity, high-variety streams of data is a fundamental, core use case.

In order to provide the results VividCortex’s customers need at a cost they can afford, we have had to invent or discover many novel techniques, some of which are patent-pending. In this book we discuss a sophisticated technique we’ve developed to select representative samples of query traffic on database servers. It uses a probabilistic data structure known as a “sketch,” among other things.


Extracting a good sample of data from a stream is much harder than it seems. There are many tradeoffs and conflicting requirements to satisfy. We are sharing our work in this book because the requirements, the approach, and the solution may be applicable to a broad variety of use cases. We would love it if we could buy a log analysis solution that used these techniques, for example. Current log analysis products are performance intensive and cost prohibitive.

Download your free copy by registering below.


Download Your Free Copy Today!