Working with alerts

When an observed metric either exceeds or falls below a threshold, the Alerts feature displays a notification in the Fastly application and sends messages to your custom integrations (e.g., Slack channels and webhooks). From the Alerts page, you can view triggered and resolved alert instances and manage your alert definitions and integrations.

Before you begin

The Alerts feature is disabled by default and can be purchased by contacting sales@fastly.com. Once enabled, any user with the appropriate roles and permissions will be able to manage alerts for Fastly services and historic stats metrics.

Limitations and considerations

The maximum number of alert definitions that can exist at any one time per customer is 50.

How alerts work

An alert definition is a configuration rule that defines when an alert should be triggered and how notifications should be handled. Specifically, it outlines:

• the observed service, data pipeline, and metric.
• the set of conditions that must be met for an alert to trigger.
• the custom destinations for alert messages (e.g., Slack channels and webhooks).

The condition set defines the threshold, the evaluation period, and whether the alert should trigger when the metric exceeds or remains below the threshold or rate. Specifically:

• A threshold is a static number or a percentage of all requests that the observed metric must either be above or below for the alert to trigger.
• The evaluation period is the number of minutes during which the observed metric is evaluated every minute to determine if the metric has gone above or remained below the specified threshold.

Alerting when a metric exceeds a static threshold

The Alerts feature supports alerting above a threshold. When creating an alert definition, you can define an above threshold based condition by setting the Trigger an alert when the metric is menu to Above. This alert will fire when the observed metric goes above the specified threshold during an evaluation period at any one minute polling interval. The alert will resolve when there are zero one minute intervals in the evaluation period that exceed the threshold.

Above a static threshold alert example

Let's say you want to be notified in your #website-alerts Slack channel when the number of 5XX responses exceed 1,000 on a single service or all your services. When configuring the alert definition, you choose whether to report on a single service or an aggregate of all services. Then, you set:

• the Data source menu to Historic Stats
• the Metric menu to status_5xx
• the Trigger an alert when the metric is: menu to Above
• the Threshold field to 1000
• the Evaluation period menu to 5 minutes
• the Integration(s) menu to website-alerts

After saving the alert definition, the system starts evaluating the observed metric in 5 minute rolling time windows (the Evaluation period). During a polling minute in one of those evaluation periods, the observed metric goes above 1,000 (the Threshold):

At the end of the fifth minute of that evaluation period, an alert is triggered because the observed metric went above the threshold during the third minute of the evaluation period. Notifications are also sent. Specifically, the Fastly application indicates that an alert has been triggered via the Alert bell icon and a message is sent to the #website-alerts Slack channel.

That alert then resolves at the end of this evaluation period because there are zero one minute intervals in the evaluation period that exceed the threshold:

Alerting when a metric is below a static threshold

The Alerts feature supports alerting below a threshold. When creating an alert definition, you can define a below threshold based condition by setting the Trigger an alert when the metric is menu to Below. This alert will fire when the observed metric remains below the specified threshold for two consecutive minutes in an evaluation period. The alert will resolve when there aren't two consecutive minutes in an evaluation period that are below the threshold. We look for two consecutive minutes in an evaluation period because network latency occasionally delays metric aggregation across POPs, and we want to ensure accuracy and limit false positives.

Below a static threshold alert example

Let's say you want to be notified in your #website-alerts Slack channel when the number of 2XX responses is below 500 on a single service or all your services. When configuring the alert definition, you choose whether to report on a single service or an aggregate of all services. Then, you set:

• the Data source menu to Historic Stats
• the Metric menu to status_2xx
• the Trigger an alert when the metric is: menu to Below
• the Threshold field to 500
• the Evaluation period menu to 5 minutes
• the Integration(s) menu to website-alerts

After saving the alert definition, the system starts evaluating the observed metric in 5 minute rolling time windows (the Evaluation period). During two consecutive polling minutes in one of those evaluation periods, the observed metric is below 500 (the Threshold):

At the end of the fifth minute of that evaluation period, an alert is triggered because the observed metric was below the threshold during the third and forth minutes of the evaluation period. Notifications are also sent. Specifically, the Fastly application indicates that an alert has been triggered via the Alert bell icon and a message is sent to the #website-alerts Slack channel.

That alert then resolves at the end of this evaluation period because there aren't two consecutive minutes in the evaluation period that are below the threshold:

Alerting when a metric exceeds a rate-based threshold

The Alerts feature supports alerting above a rate of all requests. When creating an alert definition, you can define an above rate-based condition by selecting an error rate or success rate metric to observe and setting the Trigger an alert when the metric is menu to Above. This alert will fire when the observed metric goes above the specified percentage of overall requests during an evaluation period at any one minute polling interval. The alert will resolve when there are zero one minute intervals in the evaluation period that exceed the threshold.

Above a rate-based threshold alert example

Let's say you want to be notified in your #website-alerts Slack channel when the number of 4XX responses exceeds 5% of all responses sent by a single service or all your services. When configuring the alert definition, you choose whether to report on a single service or an aggregate of all services. Then, you set:

• the Data source menu to Historic Stats
• the Metric menu to 4XX Error Rate
• the Trigger an alert when the metric is: menu to Above
• the Threshold menu to 5%
• the Evaluation period menu to 5 minutes
• the Integration(s) menu to website-alerts

After saving the alert definition, the system starts evaluating the observed metric in 5 minute rolling time windows (the Evaluation period). During a polling minute in one of those evaluation periods, the observed metric goes above 5% (the Threshold):

At the end of the fifth minute of that evaluation period, an alert is triggered because the observed metric went above the threshold during the third minute of the evaluation period. Notifications are also sent. Specifically, the Fastly application indicates that an alert has been triggered via the Alert bell icon and a message is sent to the #website-alerts Slack channel.

That alert then resolves at the end of this evaluation period because there are zero one minute intervals in the evaluation period that exceed the threshold:

Alerting when a metric is below a rate-based threshold

The Alerts feature supports alerting below a rate of all requests. When creating an alert definition, you can define a below rate-based condition by selecting an error rate or success rate metric to observe and setting the Trigger an alert when the metric is menu to Below. This alert will fire when the observed metric goes below the specified percentage of overall requests during an evaluation period at any one minute polling interval. The alert will resolve when there are zero one minute intervals in the evaluation period that exceed the threshold.

Below a rate-based threshold alert example

Let's say you want to be notified in your #website-alerts Slack channel when the Success Rate, that is, the percentage of responses with non-5XX codes, falls below 80% on a single service or all your services. When configuring the alert definition, you choose whether to report on a single service or an aggregate of all services. Then, you set:

• the Data source menu to Historic Stats
• the Metric menu to Success Rate
• the Trigger an alert when the metric is: menu to Below
• the Threshold menu to 80%
• the Evaluation period menu to 5 minutes
• the Integration(s) menu to website-alerts

After saving the alert definition, the system starts evaluating the observed metric in 5 minute rolling time windows (the Evaluation period). During a polling minute in one of those evaluation periods, the observed metric goes below 80% (the Threshold):

At the end of the fifth minute of that evaluation period, an alert is triggered because the observed metric was below the threshold during two consecutive minutes in the evaluation period. Notifications are also sent. Specifically, the Fastly application indicates that an alert has been triggered via the Alert bell icon and a message is sent to the #website-alerts Slack channel.

That alert then resolves at the end of this evaluation period because there aren't two consecutive minutes in the evaluation period that are below the threshold:

Viewing alerts

To view details for an alert that is firing or resolved, follow these steps:

1. Log in to the Fastly web interface.
2. Click Observability.
3. Click Alerts.
4. Click on an alert instance that is firing or resolved. The details page for that alert appears.

To view a list of your alert definitions, follow these steps:

1. Log in to the Fastly web interface.
2. Click Observability.
3. Click Alerts.
4. Click Alert definitions. The Alert definitions page appears.

Managing alert definitions

An alert definition is a configuration rule that defines when an alert should be triggered and how notifications should be handled.

Creating alert definitions

To create an alert definition, start by navigating to the Add alert definition page using one of these methods:

• On the Alerts page in the Firing section, click Add alert definition.
• From the Alerts page, click Alert definitions and then Add alert definition.

Next, fill out the general alert fields as follows:

1. In the Name field, enter the name of the alert.

2. In the Description field, enter a description of the alert definition.

3. Leave the All services (aggregated) value selected if you want to monitor all your services or select Single service if you want to monitor a specific service.

4. If you selected Single service, fill out the following fields:

   • From the Service menu, select a service.
   • From the Data Source menu, select a data pipeline. Data pipeline options include Historic Stats, Origin Inspector, and Domain Inspector.

Next, define the metric that you want to monitor:

1. If you're monitoring all services or selected Historic Stats from the Data source menu then, from the Metric menu, select the metric that you want to monitor.
2. If you selected Domain Inspector from the Data source menu, fill out the following fields:
   • From the Domains menu, select the domains that you want to monitor.
   • From the Metric menu, select the metric that you want to monitor.
3. If you selected Origin Inspector from the Data source menu, fill out the following fields:
   • From the Origins menu, select the origins that you want to monitor.
   • From the Metric menu, select the metric that you want to monitor.

Once you've done that, define the conditions that will trigger the alert:

If you selected an error rate or success rate alert type:

1. From the Trigger an alert when the metric is: menu, select whether the alert should be triggered when the metric is above or below a percentage of overall requests.
2. From the Threshold menu, select a percentage of overall requests that the metric must be above or below for the alert to trigger.
3. From the Evaluation period menu, select the number of minutes during which the observed metric is evaluated every minute to determine if the metric is above or below the specified threshold. Options include 5 minutes, 15 minutes, and 30 minutes.

For all other alert types:

1. From the Trigger an alert when the metric is: menu, select whether the alert should be triggered when the metric is above or below the threshold.
2. In the Threshold field, enter the number that the metric must be above or below for the alert to trigger. To set this field, you can also click and drag the threshold line on the graph.
3. From the Evaluation period menu, select the number of minutes during which the observed metric is evaluated every minute to determine if the metric is above or below the specified threshold. Options include 5 minutes, 15 minutes, and 30 minutes.

Next, optionally specify where alert notifications should be sent:

1. (Optional) From the Integrations(s) menu, select the notification channels for the alert.
2. (Optional) Click Test Alert. A test notification is sent to the notification channel you selected.

Finally, click Save Alert to save the alert. The Alert definitions page appears.

Editing alert definitions

To edit an alert definition, start by navigating to the Edit alert definition page using one of these methods:

• On the Alerts page in the Firing section, click an instance of an alert definition that you want to edit and then click Edit alert definition.
• From the Alerts page, click Alert definitions and then click the pencil to the right of the alert definition that you want to edit.

Next, update and save the alert definition:

1. Fill out the Edit alert definition fields as follows:
   • In the Name field, enter the name of the alert.
   • In the Description field, enter a description of the alert definition.
   • If the value of the Data source menu is Domain Inspector then, from the Domains menu, select the domains that you want to monitor.
   • If the value of the Data source menu is Origin Inspector then, from the Origins menu, select the origins that you want to monitor.
   • From the Metric menu, select the metric that you want to monitor.
   • From the Trigger an alert when the metric is: menu, select whether the alert should be triggered when the metric is above or below the threshold.
   • In the Threshold field, enter the number that must be above or below for the alert to trigger. To set this field, you can also click and drag the threshold line on the graph.
   • From the Evaluation period menu, select the number of minutes during which the observed metric is evaluated every minute to determine if the metric is above or below the specified threshold. Options include 5 minutes, 15 minutes, and 30 minutes.
   • From the Integration(s) menu, optionally select or remove notification channels for the alert.
2. (Optional) Click Test Alert. A test notification is sent to the notification channel you selected.
3. Click Save Alert to save the alert.

Deleting alert definitions

To delete an alert definition, follow these steps:

1. From the Alerts page, click Alert definitions.
2. Click the trash to the right of the alert definition that you want to delete. The alert definition is removed from the Alert definitions page.

Managing alert integrations

Alert integrations are external channels that receive notifications when alerts are triggered. When creating an alert definition, you specify the alert integrations that should be used. Multiple alert definitions can use the same alert integration.

Creating alert integrations

To create an alert integration, follow these steps:

Deleting alert integrations

To delete an alert integration, follow these steps:

1. From the Alerts page, click Alert integrations.
2. Click the trash to the right of the alert integration that you want to delete.
3. Click Confirm and delete. The alert integration is removed from the Alert integrations page.