An agent working at night needs more precise rules than a person surrounded by a team. It must know what to watch, when to act, and when to ask for help. In our experience with infrastructure monitoring, the decisive part is translating an operational procedure into verifiable conditions. Being available 24/7 does not mean authorization to change any system.
Separate symptom, diagnosis, and action
The first layer checks the service from outside. The second gathers data from the provider: processes, tasks, load balancer, database, and recent errors. The third evaluates whether the case belongs to a previously authorized recovery. These layers must be able to fail independently: losing diagnostic credentials should not silence the external alert.
In one implemented case, an on-call watch combines probes, AWS diagnostics, and a limited recovery of the CMS process. It does not restart the database or change DNS because of one failed response. It checks that the symptom repeats, that no deployment is in progress, that the database is not in alarm, and that it has not acted recently. An unknown cause becomes an alert with evidence.
Credentials designed for the machine
A personal session can expire while nobody is connected. Unattended operation needs machine identities and permissions limited to the necessary resources. We distinguish reading, reversible recovery, and configuration changes. The agent does not receive global administrator privileges for convenience. Rotation, revocation, and secret storage are part of the deployment.
One incident, one history
Each incident keeps its time, observed signals, diagnosis, decision, and outcome. Retries have a limit and a delay. The system records whether it acted so that several runs do not repeat the recovery. A switch suspends actions while monitoring continues. Alerts contain a summary and a reference to the evidence; they do not include credentials or dump complete personal data into a chat.
Expand autonomy with testing
We test the procedure with dependency failures, contradictory signals, and a controlled recovery. The scope can grow when those tests and experience support new cases. Rollbacks, deletions, permission changes, and business decisions need a specific policy and identified owners.
The deliverable is not just an agent switched on: it includes a runbook, an action matrix, alerts, traceability, and maintenance. That combination reduces manual monitoring and preserves control when an exception arrives.
