Skip to content
Glowo
Esc
↑↓navigate↵open⌘Jpreview
On this page

Confirmations and flapping

The settings that decide when a failing check becomes an alert — and how to stop a flapping service from paging you all night.

One failed check is not an outage. These three settings decide how much evidence Glowo needs before it changes a monitor’s status and tells anyone.

Setting Range Default Controls
confirmationsRequired 1–5 2 Consecutive failures before a failure counts
recoveryConfirmationsRequired 1–5 2 Consecutive successes before a recovery counts
minFailureDurationMs 0–300000 30000 Push monitors only: how long failure must persist before DOWN

retries sits underneath all of this: it is the number of immediate retries within a single check, before that check is recorded as a failure at all.

Each region confirms on its own

A scheduled monitor (HTTP, TCP, ping, DNS and the rest) counts confirmations per check region. A region’s vote turns to failing only after confirmationsRequired failed checks in a row from that region, and back to healthy only after recoveryConfirmationsRequired successful ones; until its run is long enough, the region abstains. The monitor follows the regions whose votes are confirmed, so one bad check in one region changes nothing.

A failure therefore takes as long to confirm as its region takes to run that many checks: about one interval for each confirmation after the first. With the round_robin or random strategy, a failure that begins just as the monitor moves to its next region is confirmed by that region’s own checks, so it can take about one interval longer.

Making recovery stricter than failure

The defaults are symmetric — two failures to go down, two successes to come back up. That is the right starting point, but it is the setting most worth raising on a service you already know is unreliable.

Going down fast is good: you want to know quickly. Coming back up fast is a trap. A service flapping between working and broken announces a recovery after the last good check, then fails again, and each cycle is another pair of notifications. Raising recoveryConfirmationsRequired to 3 or 4 asks for more evidence before Glowo declares the incident over, so one bad afternoon produces one “down” and one “up” rather than nine of each.

Raising it costs nothing but recovery latency, and a late “it’s back” is far cheaper than a false one.

Push monitors count time as well

A push monitor has no regions: your job reports to Glowo. Its confirmations count missed or failed pushes, and minFailureDurationMs counts the time since the first of them. Both must be satisfied, so with the defaults a gap that closes inside 30 seconds never pages anyone, however often the job reports.

Set it to 0 only when you genuinely want the fastest possible alert and accept the noise that comes with it. Scheduled monitors ignore it: they confirm by count, region by region.

Tuning by monitor importance

Monitor Interval Confirmations Recovery
Critical, user-facing 30–60 s 2 3
Important internal service 60–300 s 2 2
Noisy third-party dependency 300 s 3–4 3
Non-urgent background job 1800 s 1–2 2

A push monitor for a job that sometimes runs late can also raise minFailureDurationMs to a minute or two.

If you are being woken by false alarms

In order of what to try first:

  1. Raise confirmationsRequired to 3. This is the biggest single reduction in noise and costs you one extra interval of detection time.
  2. On a push monitor, raise minFailureDurationMs so a brief gap never qualifies.
  3. Add regions and use all, so a single bad network path cannot alone produce an alert. When the regions span more than one independent network, DOWN also needs the failure seen from at least two of them.
  4. Move soft failures to warn with assertions, so slowness shows as DEGRADED instead of paging.
  5. Raise retries if the failure is a genuine one-off connection reset rather than a service problem.

If a monitor is still noisy after all five, the service is probably genuinely unreliable and the monitor is telling you the truth.

Was this page helpful?