AcademyOperate

AI in IT Operations

How to build the data foundation for AI in operations, define objectives that reflect user experience, apply AI to detection, correlation and diagnosis, and grant remediation autonomy step by step with guardrails and rollback.

O201Practitioner13 slidesAbout 11 minutes

Start the course Download the PDF
1 / 13
PDF
AxiareteAcademy

O201Operate · Practitioner

AI in IT Operations

From telemetry and service level objectives to automated diagnosis and governed remediation.

FormatKnowledge pack · 13 slides
ForOperations and SRE leaders
LevelPractitioner
Axiarete Academy · O20102 / 13

The premise

Automation depends on the quality of its signals and runbooks

Noisy alerts, missing telemetry and undocumented dependencies mislead any model that relies on them. Start with clean signals, clear objectives and runbooks a machine can execute.

SignalTelemetry that describes what users experience.
ObjectiveTargets agreed with the business.
RunbookSteps precise enough for a machine to follow.
AI in IT OperationsOperate
Axiarete Academy · O20103 / 13

Foundation

The operations data foundation

TelemetryOpenTelemetry traces, metrics and logs that share service.name, so they can be joined.
TopologyServices, infrastructure and owners, with dependencies derived from traces.
ChangeDeployments, configuration changes and feature flags, with timestamps.
KnowledgeRunbooks, past incidents and known errors, in machine-readable form.
ObjectivesSLOs and error budgets for critical user journeys.
Join change data firstGoogle’s SRE book reports that “roughly 70% of outages are due to changes in a live system.” Correlate symptoms with recent deployments and configuration changes.
AI in IT OperationsOperate
Axiarete Academy · O20104 / 13

Objectives

Define reliability the way users experience it

Definitions adapted from Google’s Site Reliability Engineering books.

Service level indicatorA carefully defined quantitative measure of some aspect of the service provided, such as the share of requests that succeed.
Service level objectiveA target for an indicator over a window, such as 99.9% of requests succeeding over 28 days.
Error budget100% minus the objective. In Google’s example policy, a spent budget halts all changes and releases except P0 and security fixes.
AI in IT OperationsOperate
Axiarete Academy · O20105 / 13

AI in operations

Where AI helps, and the risks to watch

TaskWhat worksWatch for
DetectionLearned seasonal baselines for signals that have no SLOPaging on anomalies with no user impact
CorrelationGrouping alerts by topology, time and changeStale topology that groups the wrong alerts
DiagnosisQuerying logs and traces; ranking recent changes as suspectsConclusions without links to evidence
RemediationRunning approved runbook stepsWide blast radius; untested rollback
LearningDrafting summaries and reviews; clustering recurring causesDrafts from thin records; summaries in place of analysis
AI in IT OperationsOperate
Axiarete Academy · O20106 / 13

What the evidence says

Published results are strong on summaries and modest on root cause

Two production systems and one 2026 benchmark, each measured differently and not directly comparable.

51%

less time to draft each incident summary when an LLM wrote the first draft, across 300 summaries in Google’s security incident response. A person reviews every draft.

Google Security Blog, April 2024

42%

of investigations in Meta’s web monorepo had the root cause among the top five code changes its system suggested at the start, in backtesting.

Meta Engineering, June 2024

25.3%

best root-cause accuracy across five frontier agents on medium-difficulty on-call tasks, and 10.0% on hard ones, on a six-day, 50 GB testbed.

ORCA-bench, arXiv preprint, 2026

Both production teams withhold weak output: Google does not call the model when the input is under 200 tokens, and Meta suppresses low-confidence suggestions.

AI in IT OperationsOperate
Axiarete Academy · O20107 / 13

Autonomy

Grant remediation autonomy in levels

Promote each action, not each agent, on its track record at the level below.

InformThe agent reports a diagnosis with its evidence.
RecommendThe agent proposes specific runbook steps.
Execute on approvalA person approves each action before it runs.
Execute and notifyReversible, pre-approved standard changes run; the owner is notified.
AutonomousActions run within policy and error budget; people sample outcomes.
AI in IT OperationsOperate
Axiarete Academy · O20108 / 13

Guardrails

Guardrails for automated remediation

Allow-listOnly named actions on named services.
Scoped credentialsSeparate, minimal permissions for each action.
Blast-radius limitsOne instance or zone at a time.
Automatic rollbackRevert when indicators degrade after an action.
Rate limits and circuit breakersCaps per service per hour; a halt after repeated failures.
Change recordsCreated automatically for every action.
Kill switchOne control that stops all automation.
Audit trailWho or what acted, when, why and with what result.
AI in IT OperationsOperate
Axiarete Academy · O20109 / 13

Incidents

The incident lifecycle with agents

1DetectSymptom-based alerts on critical user journeys.
2TriageThe agent groups alerts and summarizes impact; a person sets severity.
3DiagnoseHypotheses linked to evidence and recent changes.
4MitigateRestore service first; preserve evidence such as logs.
5ResolveFix the cause through normal change control.
6LearnThe agent drafts the timeline; a blameless review sets actions.
AI in IT OperationsOperate
Axiarete Academy · O20110 / 13

Measurement

Measure what operations delivers

Objective attainmentObjectives met for each service, and the rate at which error budget burns.
Time to restoreReview long incidents individually; durations are skewed, so averages mislead.
Change fail rateShare of deployments that need immediate intervention, such as a rollback or hotfix.
Alert precisionShare of alerts that required action.
ToilShare of time on manual, repetitive work; Google SRE aims to keep it below 50%.
Diagnosis accuracyShare of automated diagnoses that the incident review confirms.
AI in IT OperationsOperate
Axiarete Academy · O20111 / 13

Pitfalls

Common failures in AIOps programs

Alerting on causes, not symptoms
Page on SLO burn, such as 2% of budget in an hour
Automating unreliable runbooks
Make runbooks reliable by hand first
Summaries without evidence
Link every conclusion to its source
Steering by mean time to restore
Rank incidents by error budget consumed
Administrator rights for agents
Scope credentials to each action
Skipping the incident review
Review any incident that uses over 20% of the budget
AI in IT OperationsOperate
Axiarete Academy · O20112 / 13

Getting started

A 90-day plan

Days 1–30Signal
  • Instrument the most critical services with OpenTelemetry
  • Define SLOs for five critical user journeys
  • Connect change data; record a baseline
Days 31–60Knowledge
  • Codify and test the ten most-used runbooks
  • Run agent diagnosis in shadow mode
  • Measure diagnosis accuracy
Days 61–90Autonomy
  • Add guardrails and a kill switch
  • Move three reversible actions to Execute on approval
  • Review results against the baseline
AI in IT OperationsOperate
Axiarete Academy · O20113 / 13

Summary

Key takeaways

  1. Clean telemetry, current topology and change data come before any model.
  2. Define reliability with indicators, objectives and error budgets that reflect users.
  3. Apply AI where it improves on static rules, and require evidence for every diagnosis.
  4. Grant autonomy action by action, with guardrails and automatic rollback.
  5. Measure incidents by error budget consumed, and review each significant one.
Next course · O301Building and Running AI Agents in ProductionOperate · Advanced
AI in IT OperationsOperate

Further reading

Continue learning

O301 · OperateBuilding and Running AI Agents in ProductionAxiarete AcademyAll courses and tracks

Bring this course to your leadership team

We run private working sessions that apply these frameworks to your own estate, data and pending decisions.

Request a Private BriefingExplore the Academy