How alerting works

The Alerts feature displays a notification in the Fastly control panel and sends messages to your custom integrations (e.g., Slack channels and webhooks) when a metric:

• exceeds a static threshold
• is below a static threshold
• exceeds a rate-based threshold
• is below a rate-based threshold
• increases by a certain percentage
• decreases by a certain percentage
• increases or decreases by a certain percentage

Alerting when a metric exceeds a static threshold

The Alerts feature supports alerting above a threshold. When creating an alert definition, you can define an above threshold based condition by setting the Evaluation type menu to Above (one minute). This alert will fire when the observed metric goes above the specified threshold during an evaluation period at any one minute polling interval. The alert will resolve when there are zero one minute intervals in the evaluation period that exceed the threshold.

Above a static threshold alert example

Let's say you want to be notified in your #website-alerts Slack channel when the number of 5XX responses exceed 1,000 on a single service or all your services. When configuring the alert definition, you choose whether to report on a single service or an aggregate of all services. Then, you set:

• the Data source menu to Historic Stats
• the Metric menu to status_5xx
• the Evaluation type menu to Above (one minute)
• the Threshold field to 1000
• the Evaluation period menu to 5 minutes
• the Integration(s) menu to website-alerts

After saving the alert definition, the system starts evaluating the observed metric in 5 minute rolling time windows (the Evaluation period). During a polling minute in one of those evaluation periods, the observed metric goes above 1,000 (the Threshold):

At the end of the fifth minute of that evaluation period, an alert is triggered because the observed metric went above the threshold during the third minute of the evaluation period. Notifications are also sent. Specifically, the Fastly application indicates that an alert has been triggered via the Alert bell and a message is sent to the #website-alerts Slack channel.

That alert then resolves at the end of this evaluation period because there are zero one minute intervals in the evaluation period that exceed the threshold:

Alerting when a metric is below a static threshold

The Alerts feature supports alerting below a threshold. When creating an alert definition, you can define a below threshold based condition by setting the Evaluation type menu to Below (two minutes). This alert will fire when the observed metric remains below the specified threshold for two consecutive minutes in an evaluation period. The alert will resolve when there aren't two consecutive minutes in an evaluation period that are below the threshold. We look for two consecutive minutes in an evaluation period because network latency occasionally delays metric aggregation across POPs, and we want to ensure accuracy and limit false positives.

Below a static threshold alert example

Let's say you want to be notified in your #website-alerts Slack channel when the number of 2XX responses is below 500 on a single service or all your services. When configuring the alert definition, you choose whether to report on a single service or an aggregate of all services. Then, you set:

• the Data source menu to Historic Stats
• the Metric menu to status_2xx
• the Evaluation type menu to Below (two minutes)
• the Threshold field to 500
• the Evaluation period menu to 5 minutes
• the Integration(s) menu to website-alerts

After saving the alert definition, the system starts evaluating the observed metric in 5 minute rolling time windows (the Evaluation period). During two consecutive polling minutes in one of those evaluation periods, the observed metric is below 500 (the Threshold):

At the end of the fifth minute of that evaluation period, an alert is triggered because the observed metric was below the threshold during the third and forth minutes of the evaluation period. Notifications are also sent. Specifically, the Fastly application indicates that an alert has been triggered via the Alert bell and a message is sent to the #website-alerts Slack channel.

That alert then resolves at the end of this evaluation period because there aren't two consecutive minutes in the evaluation period that are below the threshold:

Alerting when a metric exceeds a rate-based threshold

The Alerts feature supports alerting above a rate of all requests. When creating an alert definition, you can define an above rate-based condition by selecting an error rate or success rate metric to observe and setting the Evaluation type menu to Above (one minute). This alert will fire when the observed metric goes above the specified percentage of overall requests during an evaluation period at any one minute polling interval. The alert will resolve when there are zero one minute intervals in the evaluation period that exceed the threshold.

Above a rate-based threshold alert example

Let's say you want to be notified in your #website-alerts Slack channel when the number of 4XX responses exceeds 5% of all responses sent by a single service or all your services. When configuring the alert definition, you choose whether to report on a single service or an aggregate of all services. Then, you set:

• the Data source menu to Historic Stats
• the Metric menu to 4XX Error Rate
• the Evaluation type menu to Above (one minute)
• the Threshold menu to 5%
• the Evaluation period menu to 5 minutes
• the Integration(s) menu to website-alerts

After saving the alert definition, the system starts evaluating the observed metric in 5 minute rolling time windows (the Evaluation period). During a polling minute in one of those evaluation periods, the observed metric goes above 5% (the Threshold):

At the end of the fifth minute of that evaluation period, an alert is triggered because the observed metric went above the threshold during the third minute of the evaluation period. Notifications are also sent. Specifically, the Fastly application indicates that an alert has been triggered via the Alert bell and a message is sent to the #website-alerts Slack channel.

That alert then resolves at the end of this evaluation period because there are zero one minute intervals in the evaluation period that exceed the threshold:

Alerting when a metric is below a rate-based threshold

The Alerts feature supports alerting below a rate of all requests. When creating an alert definition, you can define a below rate-based condition by selecting an error rate or success rate metric to observe and setting the Evaluation type menu to Below (two minutes). This alert will fire when the observed metric goes below the specified percentage of overall requests during an evaluation period at any one minute polling interval. The alert will resolve when there are zero one minute intervals in the evaluation period that exceed the threshold.

Below a rate-based threshold alert example

Let's say you want to be notified in your #website-alerts Slack channel when the Success Rate, that is, the percentage of responses with non-5XX codes, falls below 80% on a single service or all your services. When configuring the alert definition, you choose whether to report on a single service or an aggregate of all services. Then, you set:

• the Data source menu to Historic Stats
• the Metric menu to Success Rate
• the Evaluation type menu to Below (two minutes)
• the Threshold field to 80%
• the Evaluation period menu to 5 minutes
• the Integration(s) menu to website-alerts

After saving the alert definition, the system starts evaluating the observed metric in 5 minute rolling time windows (the Evaluation period). During a polling minute in one of those evaluation periods, the observed metric goes below 80% (the Threshold):

At the end of the fifth minute of that evaluation period, an alert is triggered because the observed metric was below the threshold during two consecutive minutes in the evaluation period. Notifications are also sent. Specifically, the Fastly application indicates that an alert has been triggered via the Alert bell and a message is sent to the #website-alerts Slack channel.

That alert then resolves at the end of this evaluation period because there aren't two consecutive minutes in the evaluation period that are below the threshold:

Alerting when a metric increases by a certain percentage

The Alerts feature supports alerting when a metric increases by a certain percentage. When creating an alert definition, you can define a percentage increase condition by selecting Percentage change under the Alert type menu and selecting Increases by under the Evaluation type menu. This alert will fire when the observed metric increases by a specified percentage and by comparing the first and last minute of the Comparison period. The alert will resolve when the percentage calculation in the comparison period no longer meets the percentage condition criteria.

Percentage increase alert example

Let's say you want to be notified in your #website-alerts Slack channel when the number of 4XX responses increases by 5% on a single service excluding 4XX values less than 10. When configuring the alert definition, you choose to report on a single service. Then, you set:

• the Data source menu to Historic Stats
• the Metric menu to 4XX
• the Alert type menu to Percentage change
• the Evaluation type menu to Increases by
• the Percentage field to 5%
• the Comparison period menu to 5 minutes
• the Excluded values field to 10
• the Integration(s) menu to website-alerts

After saving the alert definition, the system starts evaluating the observed metric in 5 minute rolling time windows (the Comparison period). During a comparison period, the observed metric percentage calculation increases by 5%:

At the sixth minute and the end of the five minute comparison period, an alert is triggered because the observed metric and percentage calculation increased by at least 5% (25 - 15 / 25 = 0.4 x 100 = 40% > 5%). Notifications are also sent. Specifically, the Fastly application indicates that an alert has been triggered via the Alert bell and a message is sent to the #website-alerts Slack channel.

That alert then resolves when the percentage calculation conditions are no longer met.

Alerting when a metric decreases by a certain percentage

The Alerts feature supports alerting when a metric decreases by a certain percentage. When creating an alert definition, you can define a percentage decrease condition by selecting Percentage change under the Alert type menu and selecting Decreases by under the Evaluation type menu. This alert will fire when the observed metric decreases by a specified percentage and by comparing the first and last minute of the Comparison period. The alert will resolve when the percentage calculation in the comparison period no longer meets the percentage condition criteria.

Percentage decrease alert example

Let's say you want to be notified in your #website-alerts Slack channel when the number of 2XX responses decreases by 40% on a single service. When configuring the alert definition, you choose to report on a single service. Then, you set:

• the Data source menu to Historic Stats
• the Metric menu to 2XX
• the Alert Type menu to Percentage change
• the Evaluation type menu to Decreases by
• the Percentage field to 40%
• the Comparison period menu to 5 minutes
• the Integration(s) menu to website-alerts

After saving the alert definition, the system starts evaluating the observed metric in 5 minute rolling time windows (the Comparison period). During a comparison period, the observed metric percentage calculation decreases by 40%:

At the end of the fifth minute of that comparison period, an alert is triggered because the observed metric and percentage calculation decreased by at least 40% (60 - 100 / 100 = 0.4 x 100 = 40%). Notifications are also sent. Specifically, the Fastly application indicates that an alert has been triggered via the Alert bell and a message is sent to the #website-alerts Slack channel.

That alert then resolves when the percentage calculation conditions are no longer met.

Alerting when a metric either increases or decreases by a certain percentage

The Alerts feature supports alerting when a metric increases or decreases by a certain percentage. When creating an alert definition, you can define a percentage increase or decrease condition by selecting Percentage change under the Alert type menu and selecting Increases or decreases by under the Evaluation type menu. This alert will fire when the observed metric either increases or decreases by a specified percentage and by comparing the first and last minute of the Comparison period. The alert will resolve when the percentage calculation in the comparison period does not meet the percentage condition criteria.

Alerting when a metric exceeds a static threshold during the entire evaluation period

The Alerts feature supports alerting when all consecutive minutes within an evaluation period exceed a threshold. When creating an alert definition, you can define an above threshold based condition by setting the Evaluation type menu to Above (all minutes). This alert will fire when the observed metric goes above the specified threshold during the entire evaluation period. The alert will resolve when the conditions are no longer met.

Percentage decrease alert example

Let's say you want to be notified in your #website-alerts Slack channel when the number of 5XX responses exceed 1,000 on a single service or all your services over the entire evaluation period. When configuring the alert definition, you choose whether to report on a single service or an aggregate of all services. Then, you set:

• the Data source menu to Historic Stats
• the Metric menu to 5XX
• the Alert Type menu to Threshold
• the Evaluation type menu to Above (all minutes)
• the Threshold field to 1000
• the Evaluation period menu to 5 minutes
• the Integration(s) menu to website-alerts

After saving the alert definition, the system starts evaluating the observed metric in 5 minute rolling time windows (the Evaluation period). During an evaluation period and polling every minute, the observed metric goes above 1,000 (the Threshold) at every minute:

At the end of the fifth minute of that evaluation period, an alert is triggered because the observed metric went above the threshold during the entire evaluation period. Notifications are also sent. Specifically, the Fastly application indicates that an alert has been triggered via the Alert bell and a message is sent to the #website-alerts Slack channel.

That alert then resolves when the percentage calculation conditions are no longer met: