Published by Baron Schwartz on Jan 19, 2015 5:00:00 AM

Go's Connection Pool, Retries, and Timeouts

This is a story of intermittent 500 Internal Server errors from APIs, that ended up being caused by a hardcoded constant in Go’s database/sql package. I’ll mostly spare you the long-winded story, and get right to the problem and what we found to be the cause.

Read More
Published by Natasha Richter on Oct 27, 2014 8:50:00 AM

Debugging with Trace

When you are debugging a multi-threaded program, just adding Print() statements to your code does not always help. Coordinating several threads is not a straightforward task, and, often, the information you are printing may not be printed in the order you expect. This is especially a problem if your program is running multiple calls to the same function in different threads because there is not an inherently easy way to identify which thread printed which line. Trace solves this problem by tracking the threads.

Read More
Published by Preetam Jinka on Oct 16, 2014 3:03:00 AM

Smoke-Testing VividCortex's Agents With A Sanity Check Tool

One of our agents, vc-mysql-query, works by sniffing TCP traffic with libpcap and decoding the MySQL protocol. As you can imagine, it’s one of the most complicated portions of our codebase. It’s also difficult to test. We have a set of tests using tcpdump files of production MySQL traffic to deterministically test the sniffer code. We run the tcpdump files through the sniffer and check the generated output. The issue with this is that every time we add a new dump file, it’s a lot of manual work to know for sure what we should be seeing in the output. If the agent says that 100 SELECTs were run, how do we make sure it’s true?

Read More
Published by Baron Schwartz on Jun 23, 2014 9:06:00 AM

Discovering Query Bugs by TCP Inspection

The MySQL wire protocol includes useful result data and metadata, such as warnings and errors raised by the query. These often indicate bugs that will cause problems you may not discover for a long time, when the cleanup can be costly and damaging.

Read More

Recent Posts

Posts by Topic

see all