Bot Crashed - What Happens Now?
Learn how Botonom Office architecture handles unexpected agent crashes, auto-recovery, and what you should check first.
We start with What happens when an agent crashes?, move on to Automated self-healing stages, cover What should you check first?, and wrap up with Preventing future crashes.
What happens when an agent crashes?
When an agent encounters an unrecoverable runtime exception, Botonom's built-in watchdog service detects the failure within 5 seconds. The system immediately marks the agent status as RECOVERING and initiates the automated self-healing pipeline.
During this recovery window (typically 10–25 seconds), all in-flight tasks, webhook events, and inbound messages are safely held in durable Temporal queues. Once the agent resumes, queued tasks resume automatically with zero message loss or data corruption.
Automated self-healing stages
Botonom Office recovery pipeline executes across four rapid stages:
- Diagnosis (0–5s): The failed worker process is isolated and runtime diagnostic memory traces are written to secure operational logs.
- Warm Container Restart (5–15s): The agent's dedicated container process restarts cleanly while persistent disk storage and environment secrets remain intact.
- Context Synchronization (15–20s): Verified conversation history, active directives, and working memory facts are restored from persistent database storage.
- Queue Drain (20–25s): All tasks, external webhooks (e.g. WhatsApp, Slack), and pending user prompts queued during the downtime are executed in chronological order.
What should you check first?
In the vast majority of cases, the system recovers autonomously. If an agent remains in RECOVERING for more than 2 minutes, perform the following checks:
- Action Logs: Open Employees → Agent Detail → Action Log to inspect the latest entries marked
ERRORorTIMEOUT. - Integration Credentials: Verify whether an external OAuth token (Google Sheets, CRM, Meta Ads) expired in Skills.
- Heavy Document Ingestion: Check whether a very large file (e.g. 100+ MB uncompressed PDF) was uploaded immediately prior to the crash.
⌘Shift+R or Ctrl+Shift+R) instantly re-establishes the live connection.Preventing future crashes
Ensure smooth ongoing operations with these best practices:
- Configure Escalations: In Agent Detail → Escalations, specify fallback rules so the agent asks for human input rather than retrying conflicting operations in a loop.
- Grant Granular Skill Scopes: Only grant agents the exact skills and tools necessary for their role to prevent resource contention.
- Inspect Action Logs: Periodically review execution traces in your dashboard to catch failing third-party API keys before they cause runtime halts.