Confirmations and flapping

The settings that decide when a failing check becomes an alert — and how to stop a flapping service from paging you all night.

One failed check is not an outage. These three settings decide how much evidence Glowo needs before it changes a monitor’s status and tells anyone.

retries sits underneath all of this: it is the number of immediate retries within a single check, before that check is recorded as a failure at all.

Making recovery stricter than failure

The defaults are symmetric — two failures to go down, two successes to come back up. That is the right starting point, but it is the setting most worth raising on a service you already know is unreliable.

Going down fast is good: you want to know quickly. Coming back up fast is a trap. A service flapping between working and broken announces a recovery after the last good check, then fails again, and each cycle is another pair of notifications. Raising recoveryConfirmationsRequired to 3 or 4 asks for more evidence before Glowo declares the incident over, so one bad afternoon produces one “down” and one “up” rather than nine of each.

Raising it costs nothing but recovery latency, and a late “it’s back” is far cheaper than a false one.

Counting time as well as checks

Confirmations count checks; minFailureDurationMs counts time. Both must be satisfied.

This matters most on long intervals. With a 30-minute interval and two required confirmations, a failure already needs 30 minutes to confirm. On a 30-second interval, two confirmations is a minute — and the 30000 ms floor means a blip that resolves inside half a minute never pages anyone, whatever the interval.

Set it to 0 only when you genuinely want the fastest possible alert and accept the noise that comes with it.

Tuning by monitor importance

If you are being woken by false alarms

In order of what to try first:

1. Raise confirmationsRequired to 3. This is the biggest single reduction in noise and costs you one extra interval of detection time.
2. Raise minFailureDurationMs so brief blips never qualify.
3. Add regions and use all, so a single bad network path cannot alone produce an alert.
4. Move soft failures to warn with assertions, so slowness shows as DEGRADED instead of paging.
5. Raise retries if the failure is a genuine one-off connection reset rather than a service problem.

If a monitor is still noisy after all five, the service is probably genuinely unreliable and the monitor is telling you the truth.