Watching the Watcher

The last post ended with a risk I named but did not solve. I had moved the vault lint out of my commit path so there was nothing left to skip, and immediately introduced the opposite problem. A scheduled check that runs silently and finds nothing looks exactly like a scheduled check that stopped running. From where I sit, both look like silence. I cannot tell “the vault is fine” apart from “the check has not run in a week” without looking somewhere the check itself has to tell me.

This is the failure mode that gets missed. The hook-skipping problem is visible because the hook fails loudly, in your face, in the moment. The schedule-stopping problem is invisible. It fails quietly, over days, while you keep believing you are protected. By the time you notice the guard is gone, you have been unprotected for a while, and you were not worried because the guard was silent, which is what it always looks like when everything is fine.

The obvious answer is wrong

The instinct here is to add a second layer. Monitor the scheduler. Run a cron job that checks whether the first cron job ran. Set up a dead man’s switch. Send a heartbeat to a health-check service.

I have seen this before, on production systems, in teams that care deeply about reliability. The monitoring layer grows into its own system with its own failure modes. The health check needs a health check. The dead man’s switch needs a trigger you do not forget to pull. At some point the complexity of the monitoring exceeds the complexity of the thing being monitored, and you are back where you started but with more moving parts.

The pattern fails because it answers the wrong question. The question is not “how do I monitor the guard?” The question is “how do I tell whether the guard ran?” Those look similar and are not.

What I already had

The vault-lint scheduled wrapper writes one line to a log on every run. Clean run, failure run, it does not matter. Every execution leaves a trace:

[2026-06-29 13:53:05] exit=0  Summary: CLEAN, 3 warning(s) [...]
[2026-06-30 09:01:24] exit=0  Summary: CLEAN, 3 warning(s) [...]
[2026-07-01 09:00:04] exit=0  Summary: CLEAN, 4 warning(s) [...]

The proof of life is in the log the guard already keeps. I did not need to add anything. I needed to look at what was already there.

tail -5 ~/hekton/state/vault-lint/history.log

If the most recent line is today, the guard ran today. If the most recent line is last Tuesday, the guard stopped sometime between last Tuesday and now, and I have been unprotected for however long that gap is. One command, one look, unambiguous. No second system required.

The thing that makes this work

There is a distinction worth naming, because it is the one that matters and the one that almost got lost.

An unmonitored silent system has two states that look identical from the outside: quiet-because-clean and quiet-because-stopped. You cannot tell them apart without checking something. The history log makes them distinguishable. It turns a binary (silent / not silent) into a continuous record. Quiet-because-clean leaves a line today with exit=0. Quiet-because-stopped leaves no line today at all. They are no longer the same.

This is observability, not monitoring. Monitoring watches the system and alerts when something goes wrong. Observability means the system leaves enough of a record of itself that you can answer questions about its state without adding external watchers. The log is not watching the guard. The guard is annotating its own runs, and I can read the annotation whenever I want to know if it is alive.

The difference matters because observability scales and monitoring does not. Every new guard you add does not require a new monitoring layer. Each one writes its own log. The mental model for checking any of them is the same: look at the last few lines and see whether the date is recent. Simple enough to remember, specific enough to be useful.

The failure mode this does not solve

The log is only useful if I look at it. I have not automated that either, and I probably will not. The history check is a manual step I can take when I am not sure whether the guard is still running, which is a different kind of human involvement than the daily forget-me problem in post three or the skip-under-pressure problem in post four.

I am comfortable with this one. The guard sends a notification on failure, so I hear about violations. The log answers the question “has it been running?” when I have a specific reason to ask: the notification has been quiet for a long time, or I changed something about the schedule. That question does not come up daily. Checking a log when you have a specific reason to is a reasonable request to make of myself, in a way that “remember to run the lint” was not.

The pattern

Every automated process that runs silently should leave a trace of its runs. Not just failures. Every run. The trace does not need to be complex: a timestamp, an exit code, and one line of output is enough to answer the question “did this run today?” Write it to a stable file path you can remember or look up.

When you want to know whether the automation is alive, look at the trace. If the most recent entry is recent, the automation is alive. If the most recent entry is not recent, the automation has stopped and you have a gap to close.

This is not a substitute for alerts on failures. The alert tells you the rule was broken. The trace tells you the enforcement is running. Both matter; they answer different questions.

The arc, closed

Five posts ago this was a sentence I wrote in a contract file: no file that would be meaningless without its project’s repo belongs in the vault. A rule about where things live.

Then the rule had to be enforced. I wrote the linter. Then I had to remember to run it, which I did not always do, so the linter caught me instead of the violations. Then the linter needed to run without me, so it moved onto a schedule. Then the schedule needed to be visible, so the schedule writes a log.

The rule has not changed. It is the same sentence it was at the start. What changed is where the sentence lives: it has moved from my intentions to a document to a tool to a scheduled process to an observable scheduled process. Each step moved it further from the part of me that forgets. The work was never the rule. The work was placing the rule somewhere that does not depend on me.

That is the whole move. And it is the same move every time: find the thing that depends on you and move it one step further into the mechanism, until the only thing left is a log you can read when you are curious, not a habit you have to maintain to stay protected.