Would you know the day it broke?

Short answer: most automation fails silently, and most monitoring only catches the failure modes somebody thought of in advance. The two failures that matter are a job that stops running at all, and an alarm that reports green because nothing wrote the word it was watching for. A board that is mostly false alarms is worse than no board, because it trains people to stop looking.

Absence of output is not evidence of health

The common design checks whether a run failed. Almost nobody checks whether a run happened. Those are different questions, and the second is the dangerous one, because a job that crashed, hung, or was quietly unloaded produces no error at all. It produces silence, and silence looks exactly like a healthy quiet job.

The fix is that every scheduled job has to prove it is alive by moving something: touching a file, writing a timestamp, incrementing a counter. If that proof stops arriving inside the job's own cadence plus some grace, the silence itself is the alarm. A run counter is not good enough, because a count tells you how many times something ran and not when it last did.

The alarm has to match what the system actually writes

The second failure is subtler. Monitoring usually looks for specific strings. If the alert watches for the word ERROR and the failing component logs the word ALERT, the failure is real, visible in the log, and invisible on the board. This is not a hypothetical class of bug. It is the ordinary result of the monitor and the thing being monitored being written at different times by different people.

The check is mechanical. Take a real failure, cause it deliberately, and confirm the board goes red. If nobody has ever done that, the board's accuracy is unknown, and unknown accuracy on a monitor is the same as no monitor with extra confidence attached.

False alarms are the expensive kind

A board that is mostly noise gets ignored, and it gets ignored right before the real alarm appears. Any alert that has been red for days while the underlying job runs perfectly is not a warning, it is training. Every one of those needs to be either fixed or removed, and removing it is a legitimate answer.

The watcher needs its own watcher

The last gap is the one people find last. If the monitoring stops running, everything it reports freezes at its last value, which is usually green. So the monitor has to check its own pulse first, and when it goes quiet, its readings need to drop to unknown. Unknown is an uncomfortable status to display and it is the truthful one.

We run these four checks against our own systems on a schedule, because the bar we hold a client's system to has to be one we already meet. If you are running anything unattended right now, the useful exercise is not reading a dashboard. It is switching something off on purpose and timing how long it takes to hear about it.