On-Call Agent

On-Call Agent is a Dilijent Systems applied-AI engineering project: an incident-response agent built by our AI team to diagnose production failures and propose the fix.

The challenge

When a service starts failing in production, the person on call has to assemble the story from many separate signals: metrics, traces, container OOM records, deploys and feature flags, all while the clock runs. A language model can read that material quickly, but a model that guesses a root cause is worse than no help at all. Acting on a wrong diagnosis is expensive, and a confident answer with no evidence cannot be trusted in production.

Our solution

• A multi-model agent on AWS Bedrock AgentCore: a lead model (GLM-5.3), a fallback model (Kimi-K3, used after two failed attempts) and a cross-family reviewer (Moonshot Kimi-K3, with Claude Sonnet config-swappable)
• Tool-based telemetry instead of vector RAG: the server fetches nine live signals from Prometheus, Jaeger, Docker OOM records, GitHub deploys and feature flags
• An evidence rule enforced in code: every proposed cause must cite two or more reading IDs, and unbacked claims are rejected automatically
• A reviewer that can only reject, never invent a competing cause
• Sealed-sandbox replay of the suspected cause three times, checked against the production commit SHA hash, before anything is proposed
• The fix is opened as a pull request, never written to production directly

The result