Every other agent on this blog's self-hosting series reads data and writes a summary. A DevOps incident response agent is different: its tool list includes kubectl, cloud IAM calls, and the same infrastructure write paths a human on-call engineer uses to actually fix things. If you're going to self-host a DevOps AI agent for anything, incident response is where the case is strongest, and where the risk of getting it wrong is highest. That's a sharper version of the argument we made for self-hosting a SOC analyst agent: a SOC copilot mostly reads and summarizes alerts, but a DevOps copilot's tool layer sits one step away from executing a change against production. Hand that tool layer to a third-party API and you're not just exposing sensitive log data, you're exposing the credentials that can act on your infrastructure.
This guide covers why alert, log, and metric data belongs on infrastructure you control, what an on-call copilot actually needs to query in real time, the HolmesGPT-style ReAct architecture for building one, the latency budget a paging decision has to meet, and how to scope infra credentials so the agent can't do more damage than the incident it's investigating. It closes with GPU sizing and a cost comparison against AI-first incident platforms.
Why You Should Self-Host a DevOps AI Agent Instead of Routing Alerts Through a Third-Party API
An incident investigation prompt carries more than a stack trace. It carries internal hostnames, service topology, recent deploy history, and often the exact command output from a kubectl describe or a cloud IAM policy dump. Send that to a hosted API on a bad night and a third party has a live snapshot of your production environment during the window it's most vulnerable.
The stakes of getting incident response wrong are already high before AI enters the picture. Uptime Institute's 2026 Annual Outage Analysis found 57% of respondents said their most recent major outage cost more than $100,000, and for the second consecutive year, one in five organizations reported costs exceeding $1 million (Uptime Institute). PagerDuty's 2026 State of AI-First Operations report puts a sharper edge on the same problem: 68% of organizations lose more than $300,000 per hour during unplanned disruptions, 34% lose at least $500,000 per hour, and 8% lose more than $1 million per hour (PagerDuty). Those are exactly the minutes a hosted API adds network latency and data-residency risk to, at the moment you can least afford either.
There's a real upside case for AI in the loop too, which is why teams want a copilot in the first place. PagerDuty found 59% of organizations now actively incorporate AI into operations, and 75% of AI adopters report improved operational resilience versus 66% of non-adopters (PagerDuty). As PagerDuty CMO Katherine Calvert put it: "AI-first operations enable organizations to accelerate their incident management workflows so they can restore service more quickly during disruption" (PagerDuty). The pitch here isn't "don't put AI in the incident loop." It's "don't put AI in the incident loop in a way that hands infra credentials and production topology to a vendor you don't control."
For teams with a compliance regime that requires provable isolation, not just a promise in a deployment script, confidential GPU computing with NVIDIA TEEs extends encrypted VRAM guarantees to the inference host itself, which matters if the copilot's own logs and prompts count as regulated data.
What an On-Call Copilot Actually Needs to Query in Real Time
An on-call copilot is only as useful as the telemetry it can pull without a human relaying it by hand. That means live read access to logs, metrics, and traces, plus a retrieval layer over your own runbooks and incident history, wired into a model that can decide which source to check next instead of following a fixed script.
Logs, Metrics, Traces, and the Runbook Corpus
Four data sources feed a real investigation, and each needs a different retrieval pattern:
- Logs (Loki, Elasticsearch, CloudWatch): full-text and structured search over recent log lines, usually the first thing an agent pulls after an alert fires.
- Metrics (Prometheus, Datadog): time-series queries to correlate the alert against CPU, memory, latency, and error-rate trends before and after the incident started.
- Traces (Jaeger, Tempo, Datadog APM): distributed trace lookups to find which service in a call chain actually failed, not just which one raised the alert.
- Runbook and incident history corpus: your own postmortems, on-call runbooks, and past resolutions, embedded into a vector store so the agent can retrieve "have we seen this exact failure mode before, and what fixed it."
Our GPU monitoring for ML guide covers the DCGM and Prometheus setup that the metrics layer sits on top of if the incidents you're investigating are GPU-fleet specific rather than general service failures.
Where an Agent Helps vs Where a Human Still Decides
The clean split is investigation versus action. An agent pulling logs, correlating metrics, and drafting a root-cause hypothesis is doing work a human would otherwise do by hand across six browser tabs. An agent restarting a pod, rolling back a deploy, or scaling a service is taking an action with production consequences, and that's a different risk category entirely.
SquareOps reports teams piloting LLM-driven triage are seeing MTTR reductions of 40-70% in production, with SquareOps clients specifically reporting 50-65% MTTR reduction within 90 days of deployment and end-to-end latency from page to first hypothesis landing at 8-15 seconds (SquareOps). That range is a wide band, and the honest read is that most of the win comes from the investigation half of the job: an agent that gets a human to the right hypothesis in 15 seconds instead of 15 minutes has already captured most of the available speedup, without needing to touch a write path at all.
DORA's 2024 benchmarks give the target to measure against: elite performers resolve incidents in under an hour, high performers in under a day, medium performers in a day to a week, and low performers take longer than a week (DORA via pingfatigue.com). An AI incident-response agent should be judged against which tier it moves a team toward, not against a generic "faster" claim with no baseline attached.
Architecture: Log and Metric RAG Plus Tool-Calling LLM
A self-hosted on-call agent needs three pieces working together: a backbone model that can reason over telemetry and decide what to check next, a tool-calling loop that actually executes those checks against your observability stack, and a RAG layer grounding the whole thing in your own incident history instead of generic training data.
The Backbone Model and the ReAct-Style Tool Loop (HolmesGPT Pattern)
HolmesGPT is the clearest open reference architecture for this. It's an Apache 2.0, CNCF Sandbox project originally built by Robusta with major contributions from Microsoft, purpose-built for production observability and incident response (HolmesGPT GitHub). Its documentation is explicit about why its default toolsets are safe to run against production: "All built-in toolsets are read-only, respecting existing platform permissions (Kubernetes RBAC, Grafana roles, cloud IAM policies) with full audit logging of every tool call" (HolmesGPT docs). It integrates with 47 data sources including Prometheus, Grafana, Datadog, Loki, Kubernetes, ArgoCD, AWS, Azure, GCP, PagerDuty, and OpsGenie (HolmesGPT README), and its provider docs list Ollama and a dedicated OpenAI-Compatible category alongside hosted APIs like OpenAI, Anthropic, and Bedrock, so the backbone model doesn't have to be a hosted API either (HolmesGPT AI providers docs).
The pattern worth copying is the agentic loop itself: the model receives an alert, decides which tool to call (query Prometheus, grep a log range, describe a Kubernetes resource), reads the result, and decides the next call, repeating until it has enough context to draft a root-cause hypothesis. That's a ReAct-style loop, the same shape as any tool-calling agent, applied to observability data instead of a customer-facing task.
For a lighter first pass ahead of a full investigation loop, K8sGPT is worth pairing in front of HolmesGPT. It's a Go-based, rule-based Kubernetes scanner with over 7,700 GitHub stars that explains common failure states like CrashLoopBackOff and ImagePullBackOff in plain English, and it's strictly read-only with no write actions available (dev.to). Running K8sGPT's rule-based scan first catches the obvious, well-known failure states cheaply, so the more expensive LLM-driven investigation loop only kicks in for incidents that need actual reasoning.
Whichever backbone model you pick, its function-calling accuracy is what determines whether the tool loop is trustworthy at all. Our tool-calling benchmark guide covers BFCL v4 and tau-Bench, the right way to validate that a model reliably picks the correct tool and fills valid parameters before you trust it to call kubectl or a cloud IAM API against production. Wrapping the observability tools themselves as MCP servers is a natural fit for this layer; our GPU-accelerated MCP deployment guide covers standing up that tool-server layer and sizing GPUs for tool-call latency specifically.
Grounding in Your Own Incident History with a Vector Store
A backbone model, however capable, doesn't know your service topology, your last quarter of postmortems, or which alerts in your environment are usually false positives. That context lives in a RAG layer: your runbooks, past incident writeups, and architecture docs embedded into a vector store the agent queries before drafting a hypothesis.
Our self-hosted vector database guide covers deploying Qdrant, Milvus, or Weaviate for this layer, and the self-hosted embeddings and rerankers guide covers the TEI deployment that turns runbook text into searchable vectors and reranks retrieved matches before they hit the model's context window. For incidents that span multiple pages and require the agent to hold context across a longer investigation than a single tool call resolves, self-hosted agent memory with Mem0 or Zep is the layer that persists session state across turns, distinct from the static runbook corpus sitting in the vector store.
Before wiring any of this up to a tool that can take a write action, a guardrail layer belongs in front of the model. Our NeMo Guardrails deployment guide covers runtime rails for constraining what a model is allowed to recommend or execute, which matters more here than almost anywhere else on this blog: a hallucinated remediation step in a customer support bot is an annoyance, a hallucinated kubectl delete suggestion in an incident response workflow is a second incident.
Latency Budget for Paging Decisions
The latency budget for an on-call agent isn't a token-generation number, it's a business number: the agent's total time to first useful hypothesis has to beat how long a human takes to do the same triage manually, or the copilot isn't saving anyone anything. That's a different framing from most agent latency work, which optimizes time-to-first-token against a UX threshold. Here the threshold is a person's own baseline response time.
Why the Agent Has to Beat the Human MTTA Baseline, Not Just Be Accurate
PagerDuty's Global Incident Management Study puts median after-hours MTTA (mean time to acknowledge) at 8-15 minutes (PagerDuty MTTA study via pingfatigue.com). That's the number a real-time on-call copilot is competing against, not some abstract accuracy benchmark. An agent that takes 20 minutes to produce a correct root-cause hypothesis is technically accurate and operationally useless, because a human on-call engineer working the same page manually would have reached a conclusion faster on their own.
This is the same SLO-engineering discipline that governs any latency-sensitive inference workload, just with a different threshold to clear. Our LLM inference SLO guide covers how to instrument TTFT and P99 latency with vLLM's built-in Prometheus metrics and alert on error-budget burn in Grafana; apply that same instrumentation to the agent's own tool-call loop, not just its raw token generation, since most of an incident-response agent's wall-clock time is spent waiting on Prometheus and log queries between generations, not decoding tokens.
The alert-volume side of the equation matters just as much as raw speed. The Google SRE Workbook's benchmark for sustainable on-call load is roughly two pages per 12-hour shift (Google SRE Workbook via pingfatigue.com). An agent that pre-triages the noise before a human ever sees the page pushes real on-call load back toward that target, which is a latency win that compounds: fewer false pages means less context-switching overhead per real incident, on top of whatever time the investigation itself saves.
Track the agent's own tool-call traces the same way you'd track any production inference service. Our LLM observability guide covers Langfuse, Arize Phoenix, and Helicone for exactly this: correlating the agent's tool-call latency against the underlying GPU metrics it's competing with for the same infrastructure, which is the same monitoring pattern applied reflexively to the tool that's doing the monitoring.
Keeping Infrastructure Credentials Out of a Third-Party API
A DevOps incident response agent's tool list is the sharpest version of a problem every agent on this blog faces eventually: the more useful the tool layer, the more it can do, and the more it can do, the more damage a compromised or hallucinating agent can cause. OWASP's Top 10 for Agentic Applications names this ASI03, Identity and Privilege Abuse, describing how agents can "inherit or accumulate permissions beyond what is needed for their tasks," creating exposure through credential reuse, delegation chain abuse, or privilege escalation across agents. The guidance is direct: every agent should be treated as a distinct identity with tightly scoped permissions tied only to its specific task, and credentials should be short-lived, isolated per workflow, and never shared across agents or sessions (Indusface). That category builds directly on Excessive Agency (LLM06:2025) from OWASP's original Top 10 for LLM Applications, the risk of a model being granted more functionality, permissions, or autonomy than the task requires.
Read-Only RBAC, Scoped Service Accounts, and JIT Credentials
Three controls do most of the work here, and they're the same controls HolmesGPT's own architecture leans on by default:
- A dedicated service account, not a human's credentials. The agent should authenticate with its own Kubernetes service account or cloud IAM role, never with an on-call engineer's personal credentials or a shared admin token. Teleport's guidance on agent identity in Kubernetes environments frames this as giving each agent instance its own workload identity, using SPIFFE IDs or Kubernetes service accounts, with short-lived tokens or certificates bound to the agent's specific execution context rather than a long-lived static credential (Teleport).
- Read-only RBAC by default. Bind the agent's service account to a
ClusterRolescoped toget,list, andwatchverbs across the resources it needs to inspect, with nocreate,update,patch, ordelete. This is exactly the design HolmesGPT documents: read-only toolsets that respect existing RBAC rather than an elevated permission set carved out just for the agent. - Short-lived, task-scoped credentials for anything beyond read access. If a workflow genuinely needs a write action, gate it behind a just-in-time credential issued for that specific task and expired immediately after, not a standing write-capable role the agent holds continuously. Treat the agent as a managed non-human identity with its own audit trail, the same way you'd audit a service account, not as an extension of whichever engineer is on call that week.
The practical rule: an on-call agent's default posture is "can see everything, can change nothing." Any exception to that has to be a deliberate, logged, time-boxed grant, not a standing permission bundled into the agent's base role.
GPU Sizing for a Self-Hosted DevOps AI Agent
Size the GPU around alert volume and how many services the agent watches concurrently, not headcount. A tool-calling backbone model's weights are a fixed VRAM cost; what scales with load is the KV cache under concurrent investigations and how much context window a multi-hop trace correlation needs to hold.
| Deployment scale | Alert volume | Recommended GPU | Backbone config | VRAM used |
|---|---|---|---|---|
| Single team | Dozens of services, low concurrency | 1x A100 80GB | 8B-14B tool-calling model, BF16 | ~20-30GB |
| Multi-team | Hundreds of services, moderate concurrency | 1x H100 SXM5 80GB | 8B-14B model, FP8, higher concurrency | ~30-40GB |
| Org-wide | Hundreds of services, high concurrent investigations | 1x H100 SXM5 80GB, or 2x for headroom | 8B-14B model plus 70B-class backbone for deep correlation | ~40GB + ~70GB |
At the single-team and multi-team scale, most of the GPU's headroom goes to the RAG layer, embeddings, and vector search running colocated with the backbone, since an 8B-14B tool-calling model's weights are small relative to an A100 or H100's capacity. At org-wide scale, splitting fast triage from deeper multi-hop investigation across two GPUs keeps one long-running trace correlation from queuing behind the fast path other on-call engineers are waiting on.
Provision the instance on Spheron by logging into app.spheron.ai, selecting an A100 80GB GPU for a single-team deployment or an H100 SXM5 for higher-concurrency, multi-service coverage, and deploying with CUDA 12.4. Per-minute billing means the instance costs nothing while you're still wiring up RBAC and the observability connectors. See Spheron's docs for scripted provisioning if this is going into a repeatable deployment pipeline.
Cost: Self-Hosted Stack vs AI-First Incident Platforms
Live GPU rates as of this writing: A100 80GB PCIe runs $1.43/hr on-demand, H100 SXM5 80GB runs $3.92/hr on-demand and $2.10/hr spot.
| Deployment scale | Self-hosted GPU config | GPU + infra cost/mo | AI-first incident platform |
|---|---|---|---|
| Single team | 1x A100 80GB on-demand | ~$1,045 GPU + ~$150 infra = ~$1,195 | Typically billed per seat or per managed service, scaling with team and service count |
| Multi-team | 1x H100 SXM5 on-demand | ~$2,864 GPU + ~$200 infra = ~$3,064 | Per-seat cost multiplies across additional on-call rotations and teams |
| Org-wide | 2x H100 SXM5 on-demand | ~$5,728 GPU + ~$300 infra = ~$6,028 | Sales-quoted, scales with service count and seat count |
Infra cost covers the vector database, embedding/reranker service, and observability connectors running alongside the GPU instance. Most AI-first incident management platforms price per seat or per managed service rather than publishing a flat rate, which makes a direct comparison harder to run in public, but the shape of the tradeoff is consistent: a self-hosted GPU's cost is fixed regardless of how many services or engineers use it, while per-seat and per-service pricing scales with your team and your infrastructure footprint, the two things most likely to grow after an incident response tool actually proves useful.
Pricing fluctuates based on GPU availability. The prices above are based on 14 Aug 2026 and may have changed. Check current GPU pricing → for live rates.
The math changes if the agent's mandate expands beyond read-only investigation into taking remediation actions itself. At that point the compliance bar shifts too: an advisory copilot drafting a hypothesis for a human to act on and an agent executing a rollback autonomously don't sit in the same risk category, and the credential-scoping work above is what keeps that line from blurring by accident.
An on-call agent only earns its place if it beats how fast your team already responds, and if the infra credentials it uses never leave a boundary you control. Spheron's per-minute billing means a self-hosted incident response stack costs what it actually runs, not a per-seat rate that grows with every engineer you add to the rotation.
A100 GPU pricing → | H100 on Spheron → | View all GPU pricing →
Frequently Asked Questions
HolmesGPT is an Apache 2.0, CNCF Sandbox open-source AI agent built by Robusta with major contributions from Microsoft, purpose-built for production observability and incident response. Its built-in toolsets are read-only and respect existing platform permissions (Kubernetes RBAC, Grafana roles, cloud IAM policies), and it integrates with 47 data sources including Prometheus, Grafana, Datadog, Loki, Kubernetes, and PagerDuty. It's the clearest reference architecture for a self-hosted tool-calling incident response agent, and its provider docs support Ollama and a dedicated OpenAI-Compatible category, so the backbone model can run on your own infrastructure too.
Faster than the human baseline it's replacing. PagerDuty's Global Incident Management Study puts median after-hours MTTA (mean time to acknowledge) at 8-15 minutes. A copilot that takes 3 minutes to pull logs, correlate metrics, and draft a hypothesis is a win against that baseline even if a human still makes the final call. A copilot that takes 12 minutes to do the same thing has added latency, not removed it, regardless of how accurate its diagnosis turns out to be.
Only with scoped, read-only, short-lived credentials, and only for read paths by default. HolmesGPT's default toolsets are read-only and inherit whatever RBAC role the service account is bound to, which is the safer pattern: give the agent a dedicated service account with a read-only ClusterRole, not the credentials a human on-call engineer uses. Any write path (restarting a pod, scaling a deployment, rolling back a release) should sit behind an explicit human approval step, not run autonomously.
It depends on alert volume and how many services the agent watches, not team size. A single team monitoring a few dozen services fits comfortably on an A100 80GB running an 8B-14B tool-calling model in FP8 or BF16. An organization-wide deployment watching hundreds of services with high concurrent investigation load needs an H100 SXM5 for the throughput and context length that longer trace correlation windows require.
AI-first incident platforms typically bill per seat or per managed service, with costs scaling as you add services and users. A self-hosted A100 80GB running a tool-calling backbone model costs a fixed, predictable rate regardless of how many services or engineers use it, and the log, metric, and infra credential data involved never leaves infrastructure you control. The tradeoff is engineering time to wire up the RAG layer and tool integrations yourself instead of paying a vendor to have already done it.
