You could have the most detailed metrics displayed on your dashboard, but if no one gets notified when things break, you’re just collecting data. Alerts help turn this passive monitoring into an active response. It’s like they tell you, “Hey, your error rate just spiked!” or “Your memory usage is through the roof,” even before your users start filing support tickets, or worse, give up on your tool entirely.

By the end of this article, you will know how to set up relevant monitoring system alerts. You’ll also learn how to choose the right things to alert on, configure reasonable thresholds, and test everything before you rely on it in production.

Understanding How Alerting Works in AppSignal

AppSignal tracks a lot of metrics out of the box. On the application side of things, it focuses on error rates, response times, and request throughput. As for infrastructure, you can monitor CPU, memory, disk usage, and much more. And if there is something specific to your app you want to track, AppSignal also lets you do just that.

Metrics tend to fluctuate, and that’s normal. However, when it crosses a certain threshold, it signals there is a performance issue that can pose a risk to users. For example, your error rate going above 5% likely means something is broken.

Triggers are the rules you define in AppSignal’s anomaly detection to determine when an alert should be created. You set a trigger by choosing a metric (say, error rate), a condition (“greater than”), and a threshold value (5%).

For example: “error rate greater than 5%”

Once that threshold has been crossed, an alert opens and notifies you. When things go back to normal, it closes automatically and lets you know everything’s OK. These conditions are checked every minute. You can also add buffers like warm-up periods (to ignore short spikes) and cooldowns (to avoid repeated alerts).

AppSignal uses email by default for notifications, but it also allows other integrations for tools your team prefers, like Slack, Discord, Microsoft Teams, and more. You can also control where to get notifications based on the environment. For example, production alerts can go to the main channel, while staging goes elsewhere or remains quiet.

Decide What You Should Alert On

When it comes to alerts, there is a difference between those that have a business impact and those that are technical noise.

Alerts with business impact directly affect your users. For example, in the case where the checkout page is throwing a 500 error or your sign-up flow is broken, the users are affected, and money is potentially lost. Technical noise, on the other hand, includes metrics that might look alarming on the dashboard but don’t affect anyone. An example of this would be the CPU that briefly spikes to 80% but drops right back down during a deploy.

According to Google’s SRE handbook, the four golden signals of monitoring to focus on are:

• Latency (how fast your app responds);
• Traffic (how many requests you’re handling);
• Errors (what’s failing);
• Saturation (how much your system is being used).

AppSignal covers all four. It tracks the response times for web requests so you can set triggers when they cross a threshold. It also measures throughput (requests per minute), letting you know when there is an unexpected drop or spike that could indicate an outage or DDOS situation. Error rates and error counts for each namespace or across the whole app are monitored, too, so you can set alerts for when they exceed a percentage you’ve defined. You also get alerts before your infrastructure runs out of resources because host metrics cover CPU usage, memory usage, disk usage, and load average.

Basically, an actionable alert requires someone to do something. If an alert comes up and the correct response is to ignore it, then it shouldn’t be an alert in the first place. A good rule of thumb is to ask yourself, "If this fired at 3 a.m., would I wake someone up for it?" If the answer is no, you should either redefine the threshold or refrain from creating the alert at all.

Set up Your Monitoring System Alerts

Before you set up monitoring system alerts, make sure AppSignal is properly installed on your application. This means a language supported by AppSignal (Ruby, Elixir, Node.js, Python, or whatever language you're using) is integrated into your application, your push API key is configured, and data is showing up in your AppSignal dashboard.

Note: Make sure your servers’ time is synchronized with NTP servers. If your servers are reporting timestamps that are off, it can cause problems with triggers because AppSignal processes metrics per minute.

Setting Notification Defaults for a Namespace

Navigate to your organization's settings in AppSignal, then go to the notification dropdown, and click on Notification Defaults. Each app has namespace-level defaults for notifications.

• Select the error notification default you want, then click on Save these settings as default.
• Select the time and performance notification defaults you want.
• Click on save these settings as default.

You might set your web namespace to notify you about every error the first time it occurs since these are user-facing. In contrast, your admin namespace errors could be set to "Never notify" since those are generally low priority.

A new incident in any given namespace will automatically inherit the default settings you've configured, so you don’t have to adjust settings for each incident individually. You can also set these defaults at the organization level. That way, when a new app is added to your organization, it will automatically pick up the right notification behavior.

Creating an Anomaly Detection Trigger

Go to the Anomaly detection dropdown in your app’s navigation on AppSignal. Select Triggers and click on + Add a Trigger. Pick a metric or define a custom one you want to monitor. Next, define alert conditions, which are the threshold rules within a trigger. By doing this, you’re telling AppSignal, “Watch this metric; if it crosses this line, that’s a problem.”

For application action metrics, you can select a namespace so the trigger automatically uses the default settings you configured earlier.

You’ll also find an option for handling missing data points. By default, AppSignal expects a data point every minute. If your metric is only sent when a specific event occurs, you can enable "treat missing datapoints as zero." This ensures the alert closes properly even during quiet periods when no data comes in.

Configuring Warm-up and Cooldown Periods

Part of creating a new trigger includes configuring warm-up and cooldown periods.

The warm-up period tells AppSignal to wait for a specific amount of time before opening an alert, even after the threshold has been crossed. The threshold condition has to be met continuously for the entire warm-up period for the alert to actually fire. This helps prevent brief spikes from unnecessarily waking people up. For instance, if you set a 3-minute warm-up, the error rate has to stay above your threshold for the full 3 minutes before an alert goes off.

After an alert closes, the cooldown period prevents a new alert from being triggered immediately if the metric crosses the threshold again. This is useful for metrics that tend to fluctuate around a threshold; otherwise, you could end up with a lot of open and close notifications. A 2-minute cooldown, for example, means AppSignal won't create a new alert unless the condition is still met after those 2 minutes have passed.

Choosing Notifiers

Now you need to add a description for the trigger to provide more context when an alert is generated. Next, choose how you want to be notified. AppSignal has a default email notifier, but you can also set up custom notifiers. After you’re done with configuration, click on Save Trigger.

You can set up notifiers in Organization Settings > Notifications > Notifiers. Here, you can also add integrations from available notifiers and configure them to suit your workflow.

And yes, you absolutely can (and should) set up multiple triggers with different notifiers. For example, you might have a trigger for high error rates that sends a message to your team's Slack channel, while a separate trigger for critical host metrics goes through PagerDuty to page someone on-call. This kind of setup ensures the right people get the right alerts through the right channels.

Tune Alert Frequency and Reduce Noise

When a team receives constant notifications, and most of them turn out to be nothing, they stop paying attention. Then, when alerts on issues they should be worried about come in, they are treated like noise, just like those that came before them.

If your engineers learn that most alerts are false alarms, they'll start silencing notifications or ignoring channels altogether. At that point, it’s almost as if you had no alerts at all.

These are the situations when AppSignal’s warm-up and cooldown features, which we’ve configured earlier, come in handy, as they reduce noise in your notification channels.

Test Your Alerts Before Relying on Them

An alert that doesn't fire when it should is worse than having no alert at all because it gives you a false sense of security. And trust me, you don't want to discover an issue during an actual incident.

An easy way to test your alerts is by creating a trigger with a threshold so low that your app will cross it under normal conditions. For instance, you can set the response time threshold to milliseconds. Watch for the alert, confirm the notification lands where it should, then reset the threshold to a realistic value.

AppSignal's Anomaly Issues list page gives you an overview of all alert incidents. Open any incident to take a deeper look and study occurrence graphs, trigger conditions, recent occurrences, and a logbook where your team can leave notes.

The Time Detective tool, another one of AppSignal’s useful features, lets you see the state of your app at the exact moment the alert has fired. It's great for post-incident reviews when you need to connect the dots between an alert and everything else that was happening at the time.

Conclusion

In this article, you’ve learned how to decide what to alert on by focusing on the four golden signals. You’ve also found out why we should prioritize metrics that have real business impact, how you can create anomaly detection triggers with sensible thresholds, and why it’s important to use warm-up and cooldown periods to filter out noise. Finally, you’ve seen how you can set up notification channels so alerts reach the right people through the right tools and why it’s important to test everything before relying on it.

We’ve also got some useful resources on anomaly detection, namespaces, notification settings, and notifiers, and the AppSignal blog regularly publishes practical guides on monitoring best practices that are worth following.

Are you ready to set up the right monitoring system alerts? Get started with AppSignal!