Confirmations and flapping
The settings that decide when a failing check becomes an alert — and how to stop a flapping service from paging you all night.
One failed check is not an outage. These three settings decide how much evidence Glowo needs before it changes a monitor’s status and tells anyone.
| Setting | Range | Default | Controls |
|---|---|---|---|
confirmationsRequired |
1–5 | 2 | Consecutive failures before a failure counts |
recoveryConfirmationsRequired |
1–5 | 2 | Consecutive successes before a recovery counts |
minFailureDurationMs |
0–300000 | 30000 | Push monitors only: how long failure must persist before DOWN |
retries sits underneath all of this: it is the number of immediate retries
within a single check, before that check is recorded as a failure at all.
Each region confirms on its own
A scheduled monitor (HTTP, TCP, ping, DNS and the rest) counts confirmations per
check region. A region’s vote turns to failing only after
confirmationsRequired failed checks in a row from that region, and back to
healthy only after recoveryConfirmationsRequired successful ones; until its
run is long enough, the region abstains. The monitor follows the regions whose
votes are confirmed, so one bad check in one region changes nothing.
A failure therefore takes as long to confirm as its region takes to run that
many checks: about one interval for each confirmation after the first. With the
round_robin or random strategy, a failure that begins
just as the monitor moves to its next region is confirmed by that region’s own
checks, so it can take about one interval longer.
Making recovery stricter than failure
The defaults are symmetric — two failures to go down, two successes to come back up. That is the right starting point, but it is the setting most worth raising on a service you already know is unreliable.
Going down fast is good: you want to know quickly. Coming back up fast is a
trap. A service flapping between working and broken announces a recovery after
the last good check, then fails again, and each cycle is another pair of
notifications. Raising recoveryConfirmationsRequired to 3 or 4 asks for more
evidence before Glowo declares the incident over, so one bad afternoon produces
one “down” and one “up” rather than nine of each.
Raising it costs nothing but recovery latency, and a late “it’s back” is far cheaper than a false one.
Push monitors count time as well
A push monitor has no regions: your job reports to Glowo.
Its confirmations count missed or failed pushes, and minFailureDurationMs
counts the time since the first of them. Both must be satisfied, so with the
defaults a gap that closes inside 30 seconds never pages anyone, however often
the job reports.
Set it to 0 only when you genuinely want the fastest possible alert and accept
the noise that comes with it. Scheduled monitors ignore it: they confirm by
count, region by region.
Tuning by monitor importance
| Monitor | Interval | Confirmations | Recovery |
|---|---|---|---|
| Critical, user-facing | 30–60 s | 2 | 3 |
| Important internal service | 60–300 s | 2 | 2 |
| Noisy third-party dependency | 300 s | 3–4 | 3 |
| Non-urgent background job | 1800 s | 1–2 | 2 |
A push monitor for a job that sometimes runs late can also raise
minFailureDurationMs to a minute or two.
If you are being woken by false alarms
In order of what to try first:
- Raise
confirmationsRequiredto 3. This is the biggest single reduction in noise and costs you one extra interval of detection time. - On a push monitor, raise
minFailureDurationMsso a brief gap never qualifies. - Add regions and use
all, so a single bad network path cannot alone produce an alert. When the regions span more than one independent network,DOWNalso needs the failure seen from at least two of them. - Move soft failures to
warnwith assertions, so slowness shows asDEGRADEDinstead of paging. - Raise
retriesif the failure is a genuine one-off connection reset rather than a service problem.
If a monitor is still noisy after all five, the service is probably genuinely unreliable and the monitor is telling you the truth.