Performance monitoring is an important part of running a successful application. One of the most basic ways to tell the performance of something is to measure the duration each time it happens and distill statistics from it.
The mean or average of a collection of values is a good start to see how good or bad something behaves. It is calculated by summing all the values under consideration and then dividing by the number of occurrences.
In Ruby, this is what calculating the mean response time would look like:
def mean(array) (array.sum.to_f / array.length).round(2) end durations = [1,2,3,4,5,6,7,8,9,0] mean(durations) #=> 4.5
Note: In the example, for a more accurate result when dividing, we cast the total duration value to a Float. Otherwise, Ruby would round down to the nearest Integer, returning
Another useful statistic is the median. While it sounds similar, there’s a difference between the mean and median of a collection of values.
The median is the value separating the upper half of a set from the lower half of the set.
For a dataset with an odd number of values, you get the median by first sorting the values, then selecting the middle number. For a set with an even number of values, after sorting them, the median will be the mean of the two middle numbers.
def median(array) sorted_array = array.sort length = sorted_array.length if length.odd? # Middle number for odd arrays sorted_array[length / 2] else # Mean of two middle numbers first_value = sorted_array[length / 2] second_value = sorted_array[length / 2 - 1] (first_value + second_value) / 2.to_f end end # Even array durations = [1,2,3,4,5,6,7,8,9,0] median(durations) #=> 4.5 # Odd array durations = [1,1,2,3,4,5,6,7,8,9,0] median(durations) #=> 4
This statistic is a good way of seeing if there is a huge skew in data or a long tail.
durations = [1,2,3,4,5,2000] median(durations) #=> 3.5 mean(durations) #=> 335.83
The mean for the durations above would be
335.83 because of the single outlier of 2000ms. The median, which is only
3.5, indicates that there is a skew.
By calculating both the mean and median of a dataset, you can figure out if there are any large outliers or a long tail.
The Problem with Mean
While mean and median are good indicators of performance, they don’t tell the whole story. If you request a webpage ten times, the mean could be very low, but one or more requests can still take a very long time to complete.
The image below shows the 99th (blue) and 90th (green) percentiles and the mean (red) for a certain action in AppSignal. You can see that the 99th and 90th are quite far from the mean and there are some spikes. This means that while your average customer has a good experience, every once in a while there's a user who has to wait almost twice as long for the page to render. Ideally, you would want to get all these values as close to each other as possible, creating a more consistent experience for all your users.
For example, given the following duration set where 10 customers request a page with a duration between 100 milliseconds and 1 second.
This would result in a mean of just
190ms while one user had a very bad experience of a 1 second response time. When only tracking the mean, it's easier to think your website has great performance, while in reality every once in a while a user has a terrible experience.
The example above is only for 10 requests, but imagine if you had a thousand requests per day, that would mean a hundred of those users had a terrible experience.
To give a better idea of the distribution of the values, we use percentiles. Percentiles are similar to the median - a number that signifies a point in the dataset where half of the set is below the number and half of it is above. Percentiles are similar in the sense that the 20th percentile means that 20% of the numbers in the dataset are below that number.
Given the following (sorted) set:
If we wanted to know the 20th percentile, we can calculate it in the following way: There are 10 values in the set. The wanted value is at position 1 (
20.0 / 100 * 10 - 1) as our arrays start at zero. Since this array contains an even amount of items, we have to calculate the mean between the index (
2) and index + 1 (
3). This would result in a value of
150 for the 20th percentile.
A very naive Ruby implementation would look like this:
def percentile(array, wanted_percentile) sorted_array = array.sort index = (wanted_percentile.to_f / 100) * sorted_array.length - 1 # Check if index is not a round number if index != index.to_i sorted_array.at(index.ceil) elsif sorted_array.length.even? first_value = sorted_array.at(index) second_value = sorted_array.at(index + 1) (first_value + second_value) / 2 else sorted_array.at(index) end end # An array with an odd amount of numbers durations = [100,200,200,300,300,400,400,500,5_000] percentile(durations, 20) #=> 100 percentile(durations, 90) #=> 500 percentile(durations, 95) #=> 5000, index is a fraction, 9.5 the rounded index is 10 # An array with an even amount of numbers durations = [100,100,200,200,300,300,400,400,500,5_000] percentile(durations, 20) #=> 150, average of index 1 & 2 `(100 + 200) / 2` percentile(durations, 90) #=> 2750, average of index 8 & 9 `(500 + 5000) / 2 percentile(durations, 95) #=> 500, index is a fraction, 8.55 the index is 9
percentile function looks very similar to our
median calculation and in fact, the
median is the same as the
durations = [1,2,3] percentile(durations, 50) == median(durations) #=> true
Because of the way percentiles and averages are calculated, it’s sometimes possible to have the 90th percentile dip below the mean, for example, given the following dataset:
durations = [1,1,1,1,1,1,1,1,1,1,2000] percentile(durations, 90) #=> 1 mean(durations) #=> 182.73
This would give us a
182.73, and a 90th percentile of just
If your metric collection system only shows the 90th percentile and the mean, you’d still be able to deduce that there’s a huge outlier somewhere in your dataset if the 90th percentile drops below the average.
You are almost at 100% of this post
That's it for now! In another post, we're going to talk about how we efficiently store and calculate percentiles for all our customer's requests using Quantiles. If you have any questions or remarks about statistics and APMs, error tracking or performance monitoring, hit us up on Twitter @AppSignal or via email.