Alerting
The path from a failed check to a message in your inbox — confirmation, suppression, fan-out and retries.
A failed check does not send a message. Several things happen first, and knowing the order explains most “why didn’t I get an alert?” questions.
The path
A check fails
The probe records a failure with an error code.
Any configured retries have already been used up.
Confirmation
Glowo waits for confirmationsRequired consecutive failures and
minFailureDurationMs to elapse. Only then does the monitor become DOWN.
See Confirmations and flapping.
Suppression
The status change is dropped if the monitor is inside a maintenance window, or if the alert was recently acknowledged.
Fan-out
Every channel subscribed to that event, and in scope for that monitor, gets a delivery. Channels are independent — one failing does not stop the others.
Delivery and retry
Each delivery is recorded durably and retried on failure, with a per-channel circuit breaker. See Delivery history.
Events you can subscribe to
| Event | Fires when |
|---|---|
MONITOR_DOWN |
A monitor is confirmed down |
MONITOR_UP |
A monitor recovers |
MONITOR_DEGRADED |
A warn assertion or a response-time threshold trips |
MONITOR_CHANGED |
A DNS Diff answer changed |
CERTIFICATE_EXPIRING |
A TLS certificate is approaching expiry |
INCIDENT_CREATED |
An incident is opened on a status page |
Each channel subscribes to at least one of these. A
channel with only MONITOR_DOWN will never tell you the service came back —
a common and frustrating misconfiguration.
Why you might not get an alert
In rough order of likelihood:
- The failure is not confirmed yet. With the defaults, a monitor needs two
consecutive failures and 60 seconds before it is
DOWN. - A maintenance window is active for that monitor.
- Someone already acknowledged the alert.
- The channel is not subscribed to that event.
- The channel is scoped to specific monitors or projects, and this monitor is not among them.
- The channel is disabled.
- Delivery is failing — check delivery history for the channel.
A sane starting setup
- Email to the team, subscribed to
MONITOR_DOWNandMONITOR_UP. - Slack or Discord to your engineering channel, same two events plus
MONITOR_DEGRADED. - SMS or PagerDuty on
MONITOR_DOWNonly, scoped to your genuinely critical monitors.
The scoping in that last line is the important part. A phone that buzzes for every monitor gets silenced, and then it is not a monitoring system any more.