appsignal

New Process for Exception and Performance Incident Emails

Robert Beekman

Robert Beekman on

New Process for Exception and Performance Incident Emails

Before today, we would send an email for every Performance or Exception incident if a notification rule was triggered.

This meant that if you deployed a new version of your app and your incident notification settings were set to "first in deploy", you'd get two emails if two errors occurred.

For the notify me of "every occurrence" setting, we sent you one email per incident with a cooldown of five minutes, so as not to overflow your inbox (we found that out the hard way, back in the day 😄 ).

Incident Notification Options

Depending on the combination of apps, incidents, and these notification settings, we're sending a massive amount of emails to a subset of our users, adding a lot of email noise for these customers.

Notification changes

We've now developed a new email delivery framework to play with a couple of settings, and we're rolling out the first experiment today.

Instead of sending an email for each individual incident, we're batching notifications to (at most) one email per app, every five minutes.

For example, if 5 exceptions and 3 performance incidents are triggered after a deploy, we will send 1 email as soon as we receive the first exception from our agent. We'll collect all subsequent emails and, after 5 minutes, send them in one email.

In this scenario, you would receive the first email immediately with the errors from that agent payload batch. The other exception and performance incidents will be sent after 5 minutes.

This cuts down your email noise from 8 emails to 2 after a deploy, but still gives a quick overview of your app's state after deploying.

These changes only apply to our email notifications. Integrations such as Slack will not use this new system; you'll still get a separate message per incident.

We'd Love to Hear From You

Depending on your feedback, we can change these rules to include emails from multiple apps in an organization or tweak the time we take to collect the notifications.

Let us know what you would like to see regarding incident emails by replying to this message or reaching out to our support team.

Write for our blog

Would you like to contribute to the AppSignal blog? We're looking for skilled mid/senior-level Ruby, Elixir, and Node.js writers.

Find out more and apply

Share this article

RSS
Robert Beekman

Robert Beekman

As a co-founder, Robert wrote our very first commit. He's also our support role-model and knows all about the tiny details in code. Travels and photographs (at the same time).

All articles by Robert Beekman

AppSignal monitors your apps

AppSignal provides insights for Ruby, Rails, Elixir, Phoenix, Node.js, Express and many other frameworks and libraries. We are located in beautiful Amsterdam. We love stroopwafels. If you do too, let us know. We might send you some!

Discover AppSignal
AppSignal monitors your apps