This article is Part 1 of an ongoing series about "The Zen of System Performance." Part 2 is about how tech ops benefits from paying attention to users' perspectives. View all parts here. Baron Schwartz's presentation from MongoDB World 2017 also explores these topics — a video recording is available here.
There are two fundamental points of view for thinking about system performance. Most people tend to experience a system from only one side, and they can often analyze that perspective extremely well. This makes sense—most people are in a position where only one point of view occurs naturally. But that doesn't mean you shouldn't try to see both perspectives. Each one is equally valid, and without a complete view, you have no hope of understanding a definition of your system's performance as a whole.
Consider the parable of the blind men and the elephant. None of the men has ever encountered an elephant before. As each man circles and touches the elephant from a different angle, trying to decipher what it could be, he experiences it from a disjointed perspective. Each man fails to reach a full definition of the animal, mistaking it for something else: a snake, a tree trunk, a wall.
People who maintain only one perspective of system performance cannot possibly understand its full nature. Additionally, they see it and value it for different reasons. To understand the total performance of a system, we strongly believe you need to be able to monitor and balance both sides—and understand why each side values what it does.
The User Perspective
Users are the people who interact with a system externally and send requests for work. Every one of us is familiar with the user perspective. We're users all the time. Requests for work take many forms, depending on the system—we click, we swipe, we tap. Fundamentally, however, users are always doing the same thing: sending requests. Equally important, users are always outside the system—or at least appear to be—sending requests into it, where they no longer have visibility. This means that an individual user's experience of the system is pretty simple. They send a request; and soon (hopefully!) an answer appears. This “black box” experience is the User Perspective.
The Service Owner Perspective
On the other hand, every system has a team of engineers and administrators who manage it internally, with visibility and awareness that goes far deeper than that of the users.
The full picture of the service owners' point of view is layered and complex. Whereas a user only knows how long it takes to get a reply, the service owner sees the requests from every user; they see how the system services those requests; and, finally, they see the system's allocation of resources. This consideration of resources defines this second point of view, which we can think about as the Service Owner Perspective.
On one hand, requests. On the other hand, resources. If requests are supposed to be fast and reliable, but resources are supposed to be managed, aren't the interests of each group at odds?
A Delicate Balance
In short, the answer we've found is "no"—these perspectives only appear to contradict each other. In fact, the User Perspective and the Service Owner Perspective are deeply interconnected and affect each other constantly, in a complicated way. Truly understanding performance requires that you first successfully see both groups' perspectives and understand the tension between requests and resources. This full, holistic resolution of both perspectives is something we've begun to refer to as the zen of performance — an idea we'll continue to explore in future posts.
In the meantime, keep learning about designing systems while thinking about performance, with a free recording of Preetam Jinka's webinar, "How to Be a Performance-Driven Engineer."