Unattended agents that explain themselves
· 4 min read
An unattended agent runs when nobody is at the desk, so I design each one to explain itself. When the operations team logs in, they can see what ran, what each subagent decided, and what was handed to a person. If something doesn't look right, the engineering team can open the audit trail, find the exact item and step, and fix it quickly.
These guardrails give the customer visibility every day, and they make triage calm and quick when it's needed.
Every run writes the same audit trail the engineering team uses for triage, so finding the cause starts with a lookup, not a guess.
1. Let the orchestrator schedule around the business
The orchestrator owns the schedule, and every schedule answers one question: when does the business need this done? At a regional bank, mobile deposits came in seven days a week, so that agent ran five times a day, every day. Merchant deposit exceptions were handled during the hours the operations team could act on them. The duplicate item report ran at 8 AM on weekdays so it was ready when the team started, and underwriting ran twelve hours a day on weekdays to keep up with requests.
2. Make every run safe to repeat
Runs can stop partway: a session times out, a system goes down for maintenance, a machine restarts. Each item carries a status, subagents skip items that are already finished, and nothing is posted to a system of record twice. Recovery is then a simple rerun, which keeps things calm for everyone.
3. Write an audit trail for every item and step
The most useful question after any run is "why did this item end up here?" So every step a subagent takes writes an audit trail entry: what it saw, which rule applied, what it did, where the item went next, and when. When an item needs a person, the agent sends it for human review with that context, and the reviewer can resume the agent once they've made the call. For batch work, the entries roll up into an audit trail per batch. The same record answers a customer's question and gives the engineering team the starting point for triage.
import json, logging, time
trail = logging.getLogger("audit_trail")
def audit(item_id, step, inputs, rule, outcome):
trail.info(json.dumps({"item": item_id, "step": step, "inputs": inputs,
"rule": rule, "outcome": outcome, "ts": time.time()}))
def run_batch(items, policy, subagent, notify):
step, start, last = subagent.__name__, time.time(), None
counts = {"done": 0, "to_person": 0, "failed": 0}
try:
for item in items:
if item.status == "done":
continue # safe to rerun
last = item.id
try:
outcome, rule = subagent(item, policy)
counts["done" if outcome == "posted" else "to_person"] += 1
audit(item.id, step, item.summary(), rule, outcome)
except Exception as e:
counts["failed"] += 1
audit(item.id, step, item.summary(), None, f"error: {e}")
finally: # sent even if the run stops
notify({"minutes": round((time.time() - start) / 60, 1),
"stopped_at": (last, step), **counts})
return counts
4. Share what happened
A short summary the business already reads goes a long way. The duplicate item agent produced a daily PDF for the deposit operations team. The document pipeline I rebuilt sent a summary email after each run: how many items were processed, which ones went to a person and why, and anything that failed.
5. Rerun evals before every change
Most surprises come from change: a core system upgrade, a new screen layout, a new rule. Before any workflow change goes live, and after every system upgrade, I rerun an eval set of known items and check each result against its expected outcome. If they all match, the change ships. If one doesn't, the audit trail of the eval run shows exactly which step behaves differently.
6. Triage the same way every time
We usually spotted an issue in two places: the audit trail, and the email notifications that showed how long each agent ran and where it stopped. The email went to the engineering team, and if a run stopped overnight, the engineer who owned that client picked it up the next morning. From there the steps don't change: find the item and step in the audit trail, fix the config or the subagent, rerun the eval set, then safely rerun the unfinished items. Because every piece is recorded, the customer can follow what happened and the engineering team can resolve it quickly.
- Four deposit operations agents at a regional bank each had their own schedule, from five short runs a day to a twelve-hour weekday shift, set around when the operations team needed results.
- Merchant deposit corrections over $10,000 always went to a person, and every batch kept its own audit trail.
- Outside banking, a collections engine for a legal services customer processed 12,000+ records in about 84 seconds, and 99.6% of them were handled correctly with no manual fix.
Other ways teams approach this
Every team monitors agents a bit differently, and there's no one right answer. Some use the tracing tools they already run for other services, so model calls and tool use show up next to everything else. Others build dashboards inside their agent platform. Regulated customers generally expect ongoing monitoring of anything that makes decisions. Audit trails for every step, a short daily summary and an eval set you rerun before changes are a simple combination that works across most setups.
A checklist for any unattended agent
- The orchestrator's schedule matches when the business needs results.
- A rerun after a failure is always safe.
- Every item and step has an audit trail entry, and items that need a person go to a reviewer who can resume the agent.
- The business gets a short summary after each run.
- An eval set is rerun before every change.
- Email notifications show how long each run took and where it stopped.