Before today, we would send an email for every Performance or Exception incident if a notification rule was triggered.
This meant that if you deployed a new version of your app and your incident notification settings were set to "first in deploy", you'd get two emails if two errors occurred.
For the notify me of "every occurrence" setting, we sent you one email per incident with a cooldown of five minutes, so as not to overflow your inbox (we found that out the hard way, back in the day 😄 ).
Depending on the combination of apps, incidents, and these notification settings, we're sending a massive amount of emails to a subset of our users, adding a lot of email noise for these customers.
Notification changes
We've now developed a new email delivery framework to play with a couple of settings, and we're rolling out the first experiment today.
Instead of sending an email for each individual incident, we're batching notifications to (at most) one email per app, every five minutes.
For example, if 5 exceptions and 3 performance incidents are triggered after a deploy, we will send 1 email as soon as we receive the first exception from our agent. We'll collect all subsequent emails and, after 5 minutes, send them in one email.
In this scenario, you would receive the first email immediately with the errors from that agent payload batch. The other exception and performance incidents will be sent after 5 minutes.
This cuts down your email noise from 8 emails to 2 after a deploy, but still gives a quick overview of your app's state after deploying.
These changes only apply to our email notifications. Integrations such as Slack will not use this new system; you'll still get a separate message per incident.
We'd Love to Hear From You
Depending on your feedback, we can change these rules to include emails from multiple apps in an organization or tweak the time we take to collect the notifications.
Let us know what you would like to see regarding incident emails by replying to this message or reaching out to our support team.
