# Watchdogs that stay silent or mail too much

> A watchdog fails quietly in two ways: it says nothing when a source dies, or it mails so often that nobody reads it. Where the clock, limit and mail go wrong.

Published 2026-10-28 · https://aiskills402.com/blog/watchdog-alerts-silent-and-noisy

A source that has stopped answering produces no error, no stack trace and no red line anywhere. A feed that went quiet, a publisher that stopped publishing and a job that stopped running all look exactly like a calm day. The watchdog is therefore the only symptom such a failure will ever have, and it can go wrong in two opposite directions. Either a genuine death goes unreported, or healthy sources trigger so many messages that the recipient stops opening them. Both come from a small set of design mistakes, mostly in how silence is measured, how long a limit is, and how the message is written. The sections below take them one at a time and finish with a short test that can fail.

## Why both failures cost the same

The first failure is obvious: nobody hears about the dead source. The second is slower and just as damaging. The Google SRE book describes what frequent alerts do to people: they second-guess them, skim them, or ignore them, sometimes brushing past a real one that is buried in the noise, and it asks for alerting with a strong signal and very little noise ([Google SRE book](https://sre.google/sre-book/monitoring-distributed-systems/)). A watchdog that raises the alarm over healthy sources trains its reader to skip the single message that matters.

The Prometheus documentation points the same way in practical terms: alert on symptoms, keep the number of alerts small, and leave slack so that small blips do not page anyone ([Prometheus alerting practices](https://prometheus.io/docs/practices/alerting/)). Each mistake further down breaks at least one of those ideas.

## Mistake one: measuring silence with the wall clock

The tempting formula subtracts the last success from the clock on the wall. It works until something pauses whatever does the asking: a release freeze, a stopped scheduler, an account that has used its allowance. When asking resumes, every source shows an old success, so all of them seem dead together, and the single genuine death is lost among the false ones.

The opposite slip is just as bad. A source that was skipped in this cycle, for instance because its quota was spent, but still has its "checked" stamp moved forward, looks alive for ever.

The repair is to keep two timestamps per source. One is updated when the source is truly approached, the other when it truly replies. Silence is the distance between them, and that distance stands still while nobody is asking. The ordinary clock keeps one narrow job: deciding whether enough time has passed since the previous mail.

## Mistake two: an alarm that exists only inside a band

To avoid repeating itself without storing anything, some watchdogs fire only while the silence sits within a band just above the threshold. The hope is that each source passes through the band once and so triggers a few mails at most. Two things go wrong. Silence measured from the last attempt freezes whenever nobody attempts, so the band stays true and the mail repeats on every tick. And a source that died long ago is already past the band, so it is never reported at all.

The fix is to split the questions. The verdict on death needs no upper band. The decision to send a reminder looks at ordinary time and at a stored note, per lamp (one watched source and its status row), of when the last mail went out, so a daily reminder is possible.

## Mistake three: a limit that matches the schedule

A limit has to exceed the largest gap that the watched thing may legitimately leave, plus one tick of the watchdog itself, plus some slack. A publisher that runs once a day, watched with a limit of one day, makes the watchdog measure the calendar instead of the pipeline: it reports a death every morning just before the publisher runs.

Prometheus gives a concrete rule for batch jobs. The alert should fire when the job has not succeeded recently enough, with a threshold that leaves room for at least two complete runs. Its example is a four-hour schedule with a one-hour runtime, where 10 hours is the sensible limit ([Prometheus alerting practices](https://prometheus.io/docs/practices/alerting/)). The same page adds that if a single failed run is unacceptable, the job should run more often, because one failure should not need a human.

Healthchecks.io builds the idea into its settings: a check has an expected period between pings and a separate grace time, the extra wait before an alert for a late check ([Healthchecks.io documentation](https://healthchecks.io/docs/configuring_checks/)). Grace is a first-class setting there because lateness within normal variation is expected.

Scheduler behaviour belongs in the choice of the number too. Cloudflare states that Workers started by a cron expression run on underutilized machines ([Cloudflare Cron Triggers](https://developers.cloudflare.com/workers/configuration/cron-triggers/)), a reason not to size a limit to the exact tick. A related trap is a guard placed ahead of the step that resets the quantity it measures: it always observes the gap at its widest, so it needs the same extra slack.

## Mistake four: a lamp that is born already dead

Some designs write a status row only when the first failure happens, since healthy ticks write nothing. If that new row begins with an empty or zero value for the last success, the judge reads it as "this source never worked" and sends mail immediately, although the source was fine a minute earlier. A second variant is a new failure episode that carries over the success time from an earlier episode.

A new episode should take the previous heartbeat as its starting point, meaning the latest moment when the periodic job is known to have executed. While an episode is running, that value stays untouched. An empty value is tolerable only if the heartbeat is unreadable, and then the situation deserves a loud report instead of passing as normal.

## Mistake five: checks that sit behind the brake

Two pieces of code decide whether the watchdog can see anything at all: the probe that writes health rows, and the sweeper that forgets sources which were removed on purpose. Both are often placed after an early exit such as a daily cap, an allowance or a brake meant to protect an expensive step.

- A probe behind the brake stops asking while the brake holds. A source that dies in that period remains unseen until the brake is released, which may take days. Only the costly part, such as a paid model call, should wait behind the gate; the cheap pull belongs ahead of every gate.
- A sweeper behind the brake is reached only when the day is quiet, so a source retired deliberately keeps ageing and alarming. It belongs as the first step of each tick, before any early exit can skip it.

## Mistake six: the mail itself

Several faults live in the message and in its memory.

- **A failed send recorded as delivered.** If the note that an alert went out is saved when the send was merely tried, one failure silences all later ticks. Some mail services return an error code instead of raising an exception, so the result has to be inspected. When that note cannot be read, the safe choice is to send, because a repeat is audible and silence is not.
- **Memory deleted by age.** The record of when a lamp was last mailed scales with the number of lamps, not with elapsed time. Deleting it by age can remove precisely the record that prevents repeats. It should go away together with its lamp.
- **A subject that names nothing.** A subject such as "alert" cannot distinguish one dead source from a stopped pipeline, and a phone shortens long subjects at the tail. Put the environment word first, then the source.
- **Environment guessed from one signal.** If an absent setting defaults to the production label, a staging copy introduces itself as production. Production should be claimed only if two unrelated signals both say so. Healthchecks.io lets a check carry an environment tag for this kind of separation ([Healthchecks.io documentation](https://healthchecks.io/docs/configuring_checks/)); in a hand-built mail, a plain word at the front of the subject does the job. Square brackets are a poor choice, since at least one large mail service has been seen to strip them, which is an observation from practice and not something the pages cited here state.

## Mistake seven: a lamp with no writer, and a monitor inside what it watches

When a scheduler is replaced, the job that wrote a heartbeat may be left behind. The guard keeps reading the row, nothing updates it, and the guard declares a false death or simply sleeps. This is easy to miss, because on a good day the heartbeat shows nothing. Build the list of jobs by reading the old scheduler itself rather than recalling it, then search what runs today for the name of each lamp.

Finally, a monitor whose goal is to learn that the service is reachable cannot live entirely inside that service. If the probe, the heartbeat, the guard and the mail all run in one account, whatever takes that account down also takes the monitor down, and the resulting quiet is easy to mistake for good health. The Prometheus documentation says the same about its own stack: be confident that monitoring works, and supplement internal checks with a monitor from outside, which also works when the internal systems break entirely ([Prometheus alerting practices](https://prometheus.io/docs/practices/alerting/)).

## A check that can fail

Take any watchdog and run four experiments, each of which has to be able to come out red.

1. Stop the scheduler for longer than the limit, then start it again. If every source is reported dead, the clock is wrong.
2. Kill one source and wait through several repeat intervals. Count the mails: more than one per interval means repetition is not remembered, zero means the band or a gate swallowed it.
3. Make the mail service answer with an error. A second attempt must happen on the next tick.
4. Start a copy in a test environment with the production variable missing. Its subject must not claim production.

An experiment that cannot produce a bad result proves nothing, so run each once against a deliberately broken version before trusting a green outcome. For the broader habit of testing a report instead of believing it, the article on [what an agent claims against what its record shows](/blog/agent-says-done-is-it) applies the same attitude to agent summaries.

## When this does not apply

The topic here is staleness and heartbeat alarms for sources and jobs that should keep answering. Threshold alerting on latency or errors, paging policy, on-call rotation and tooling choice are outside it. The cited pages are general documentation from their authors and support the points about noise, symptoms, grace time, batch thresholds and external monitoring. The specific mistakes (two timestamps, the band, a status row that appears with the first failure, a note purged by age) are design practice drawn from real incidents and review, not from those pages, and no new experiment was run for them. A system with a single source, a person watching a dashboard all day, or a tolerance for very late discovery may reasonably skip several of these. Mail-service behaviour such as the handling of brackets in subjects can change, and this guide did not check it again.

## Reviewing a watchdog before you rely on it

If a second look at a watchdog would help, the [watchdog-alert-review](https://aiskills402.com/skills/watchdog-alert-review) skill takes code or a design description for a liveness check, an alarm on stale data or a daily heartbeat mail. It reports where the alarm would stay quiet over a dead source or fire over a healthy one: how silence is clocked, the band, the limit against the cadence, a new row born at zero, probe and sweeper placement, failed sends, purged memory, subjects and environment stamps, lamps that nothing writes any more and monitors that live inside what they watch. Each finding names where, why and how to repair it, followed by one verdict, and a clean paste is answered with one short sentence.