Incident Lifecycle

​

Overview

kubectl get iss -A          # columns: Severity, State, Risk, Age
kubectl get rp -n <ns>      # columns: Issue, Attempt, State, Age

​

Incident States

​

State Machine Flow

Detected → Analyzing → Remediating ──(plan Completed)──────────────→ Resolved
               ↑            │                                          ↑
               │            ├─(plan Completed with containment)→ Contained ─┘ (human restores replicas)
               │            │
               │            └─(plan Failed / RolledBack)
               │                     │
               └── re-analysis ←─────┤ attempts < max
                   (with failure     │
                    context)         └─ attempts = max → Escalated
                                                            │
                                         auto-resolve if the resource
                                         recovers (configurable) → Resolved

​

Detection Phase (Detected)

1. Existing incident — if a non-terminal Issue (anything except Resolved, Escalated, Failed) already exists for the same resource (kind + name + namespace), the anomaly is attached to it and its risk score is raised if the new score (which counts the anomaly just attached) is higher
2. Resolution cooldown — if an Issue for the same resource was Resolved within resolutionCooldownMinutes, the anomaly is suppressed
3. Noise reduction — repetitive, seasonal, flapping or high alert-fatigue (score > 80) anomalies are suppressed
4. Signal scoring — each signal type has a weight (oom_kill=40, error_rate=30, pod_restart=25, deploy_failing=25, latency=20, pod_not_ready=20, cpu_high=15, memory_high=15, any other signal=10); the risk score sums all uncorrelated anomalies for the resource in the last 10 minutes, capped at 100
5. Severity determination — oom_kill is always critical; otherwise Critical (risk ≥ 80), High (≥ 60), Medium (≥ 30), Low (below 30)
6. Incident ID — format INC-YYYYMMDD-NNN, stored in the label platform.chatcli.io/inc-id. The Issue name is <resource>-<signal>-<unix-timestamp> (e.g. payment-api-oom-kill-1773900000); the REST API and kubectl address Issues by that name, not by the INC ID
7. Max remediation attempts — set on the first reconcile from aiops.maxRemediationAttempts (default 5). If a runbook is later selected, its maxAttempts replaces this value (see the tip below)

kubectl get iss -A -l platform.chatcli.io/inc-id=INC-20260319-001

​

Analysis Phase (Analyzing)

1. Runbook candidate discovery (tiered):
   • Tier 1: runbooks matching SignalType + Severity + ResourceKind
   • Tier 2: runbooks matching Severity + ResourceKind with a different signal
   • Multiple runbooks can exist per trigger (different root causes produce different runbooks)
2. AI validates candidates: the LLM receives all candidate runbooks and evaluates each against the current root cause analysis:
   • RUNBOOK_APPROVED: <name> → uses that runbook (fast path); if the name does not match any candidate, the first candidate is used
   • RUNBOOK_REJECTED → skips all candidates, uses AI suggestions or agentic mode
   • Neither → uses the first candidate as default (backward compatibility)
3. If no runbook was selected and the AI suggested actions → generates a new runbook from those actions and uses it
4. If no runbook and no AI actions → enters Agentic Mode (AI-driven step-by-step)
5. Creates the RemediationPlan <issue>-plan-<attempt> and transitions to Remediating

​

Remediation Phase (Remediating)

1. Pre-flight snapshot of the target workload captured for rollback
2. For each action in the plan:
   • OBSERVE — from the second action on, checks if the resource is already healthy. If so, stops immediately without executing the remaining actions (early exit)
   • ACT — executes the action and records a checkpoint
   • If the action fails → automatic rollback to the pre-flight snapshot
3. Final health verification (polls every 10 seconds, for up to 90 seconds; on timeout, rollback to the pre-flight snapshot)
4. On success → Resolved (or Contained, if a containment action ran) + PostMortem generated
5. On failure → re-analysis with the failure context (next attempt) or escalation

Example: plan with 3 actions (AdjustResources + DeletePod + RollbackDeployment)

  Action 1: AdjustResources → SUCCESS (memory 1Mi → 64Mi)
  Action 2: OBSERVE → resource healthy (ReadyReplicas == Desired)
            → EARLY EXIT! Skips DeletePod and RollbackDeployment
            → Evidence: "Resource healthy after 1/3 actions — skipped remaining 2 actions"

​

Retry Mechanism

• Attempt < max attempts: the failure evidence of all failed plans is written to the AIInsight (annotation platform.chatcli.io/failure-context), the analysis is cleared, and the Issue goes back to Analyzing — potentially selecting a different runbook or strategy
• All attempts exhausted: transitions to Escalated

​

Escalated State — What Operators Must Do

1. Starts the first matching enabled EscalationPolicy (by severities; a policy with defaultPolicy: true is the fallback) and notifies level 0 — skipped for chaos-induced Issues
2. Sends the notifications of any NotificationPolicy rule that matches the Escalated state
3. Advances to the next level when the current level’s timeoutMinutes expires, up to the last level (see Notifications)
4. Records an issue_escalated audit event
5. Keeps checking the resource every 30 seconds for auto-resolve (below)

1. Acknowledge the incident (records who is on it and stops the escalation):
   curl -X POST "https://operator:8090/api/v1/incidents/payment-api-oom-kill-1773900000/acknowledge?namespace=production" \
     -H "X-API-Key: $API_KEY"
   
   Acknowledging writes the annotations aiops.chatcli.io/acknowledged, -at and -by (the -by value is the caller’s API-key role, not a person; the request body is ignored). The escalation stops at its current level: no further level and no repeat, and the acknowledgement is recorded on the EscalationPolicy’s status.activeEscalations entry. Snooze (/snooze, body {"duration": "1h"}, a positive duration) holds every notification except Resolved, and the escalation, until aiops.chatcli.io/snoozed-until; a held escalation page goes out when the snooze ends and the level’s timer restarts. See Notifications.
2. Investigate and fix the issue manually
3. Resolve the incident via one of three methods: Method 1: REST API (recommended for automation/scripts; operator role)
   curl -X POST "https://operator:8090/api/v1/incidents/payment-api-oom-kill-1773900000/resolve?namespace=production" \
     -H "X-API-Key: $API_KEY" \
     -H "Content-Type: application/json" \
     -d '{"resolution": "Fixed memory leak in payment-service v2.4.1, deployed hotfix manually"}'
   
   Without ?namespace=, the first Issue with that name in any namespace is used. The call returns 409 if the Issue is already Resolved. It sets the status and the annotations aiops.chatcli.io/resolved-by (the role), resolved-at and manual-resolution: "true". Method 2: Web Dashboard Navigate to the incident detail page and click the “Resolve” button. Enter the resolution note in the prompt (optional). Method 3: Kubernetes Direct (advanced) — status is a subresource, so the patch must target it:
   kubectl patch issue payment-api-oom-kill-1773900000 -n production --subresource=status --type=merge \
     -p '{"status":{"state":"Resolved","resolution":"Manual fix applied"}}'
   

​

Auto-Resolve for Escalated Issues

“Auto-resolved: resource recovered while awaiting human intervention”

• An operator fixes the issue manually (kubectl rollout undo, etc.) without using the API
• The resource self-heals (e.g., a transient network issue resolves)
• A CI/CD pipeline deploys a fix while the incident is still open

​

Configurable AIOps Parameters

apiVersion: platform.chatcli.io/v1alpha1
kind: Instance
metadata:
  name: chatcli-prod
spec:
  provider: OPENAI
  model: gpt-5.4
  aiops:
    maxRemediationAttempts: 5     # default: 5, range: 1-10
    resolutionCooldownMinutes: 10 # default: 10, range: 0-120
    dedupTTLMinutes: 30           # default: 30, range: 5-1440
    enableAutoResolve: true       # default: true
    agenticMaxSteps: 10           # default: 10, range: 3-30

​

Remediation Plan States

​

Agentic Remediation Mode

1. AI proposes an action via the AgenticStep RPC
2. Action is executed and the result is observed
3. AI analyzes the observation and proposes the next action
4. Loop continues until resolved or a guardrail stops it

• Max steps: 10 (configurable via aiops.agenticMaxSteps)
• Max time: 10 minutes per agentic plan (the convergence detector already stops it at 8 minutes)
• Convergence detection (counted in chatcli_operator_agentic_convergence_stops_total):
  • Last 3 observations identical → force stop
  • Alternating A→B→A→B action pattern → force stop
  • Last 5 actions all failed → force stop

​

Decision Engine Confidence Thresholds

​

Rollback Engine

1. Pre-flight snapshot — captured before ANY action. Automatic rollbacks always restore this snapshot of the target workload.
2. Per-action checkpoints — a snapshot recorded before EACH action (for node actions, a node snapshot). They are kept in status.actionCheckpoints for audit and the PostMortem timeline; they are not replayed automatically (there is no automatic partial rollback).

• Action execution fails
• Health verification times out (90 seconds)

• Deployment: replicas, container images, container resources (and HPA min/max)
• StatefulSet: replicas, images, resources, partition
• DaemonSet: images, resources, max unavailable
• Job/CronJob: suspend, deadline, backoff limit, parallelism
• Node: schedulable state — supported by the engine, but because automatic rollback restores the workload snapshot, a CordonNode/DrainNode is not undone automatically (use UncordonNode)

​

Remediation Action Types

​

Deployment and cluster-level (19 actions)

​

StatefulSet (9 actions)

​

DaemonSet (7 actions)

​

Job (9 actions)

​

CronJob (10 actions)

​

Runbook Learning System

​

Node Failure — Remediation Flow

Node MemoryPressure detected
  → Anomaly CR created (signal: memory_high, resource kind: Node)
    → Issue created (severity from the risk score; memory_high alone = low)
      → AI analyzes: "Node worker-2 with MemoryPressure, echo-app pods impacted"
        → Remediation: CordonNode (prevent new pods) + DrainNode (remove existing pods)
          → Kubernetes re-schedules pods on healthy nodes
            → Health verification of the target (see the note below)

​

How Runbooks Are Named

auto-{signal}-{severity}-{kind}-{hash}

Examples:
  auto-oom-kill-critical-deployment-a3f2b1  (cause: tail /dev/zero)
  auto-oom-kill-critical-deployment-c7d4e9  (cause: memory limit too low)
  auto-pod-not-ready-low-deployment-e8b3d2  (cause: bad image tag)

​

Multi-Runbook Selection

New OOMKill incident on Deployment
       ↓
3 candidate runbooks found (different root causes)
       ↓
All 3 injected into AI context with their steps and descriptions
       ↓
AI analyzes current root cause and responds:
  "RUNBOOK_APPROVED: auto-oom-kill-critical-deployment-c7d4e9"
  (because this incident is caused by low memory limits, matching that runbook)
       ↓
Selected runbook executed → fast resolution without agentic loop

​

Runbook Lifecycle

​

PostMortem Generation

• Timeline — chronological events from detection to resolution
• Root cause, summary and impact — from the AI analysis
• Actions executed — complete remediation history
• Lessons learned and prevention actions — AI recommendations
• Git correlation and GitOps context — recent changes that may have caused the issue
• Cascade chain — related incidents across services
• Trending — recurrence of similar incidents

​

PostMortems with requiresHumanAction

# Issue
kubectl get issue <name> -o jsonpath='{.status.requiresHumanAction}'
# true
kubectl get issue <name> -o jsonpath='{.status.requiredAction}'
# restore the deployment's replicas to the desired count after fixing the root cause...

# PostMortem
kubectl get postmortem pm-<name> -o jsonpath='{.status.requiresHumanAction}'
# true
kubectl get postmortem pm-<name> -o jsonpath='{.status.requiredAction}'
# (same text)

• If a PostMortem with requiresHumanAction: true is set to Closed without the annotation aiops.chatcli.io/human-action-acknowledged set to a truthy value (true, True, yes, ack, acknowledged), the PostMortemReconciler reverts it to Open (even after a forced kubectl patch)
• When auto-resolve fires (a human restored the replicas), the controller clears both fields on the Issue and sets the condition RequiresHumanAction: False. The PostMortem keeps its flag until the human action is acknowledged
• The REST API exposes the fields as top-level fields of the incident item (requiresHumanAction, requiredAction, returned under both spec and status of the response), so dashboards render them without fetching the PostMortem

# Via REST API (namespace defaults to "default" when omitted)
curl -X POST -H "X-API-Key: $API_KEY" -H "Content-Type: application/json" \
  "$AIOPS_URL/api/v1/postmortems/<pm-name>/ack-human-action?namespace=default" \
  -d '{"acknowledgedBy":"sre-team","note":"rolled back to v1.2.3 and scaled to 3 replicas"}'

# Or via kubectl
kubectl annotate postmortem <pm-name> -n default \
  aiops.chatcli.io/human-action-acknowledged=true \
  aiops.chatcli.io/human-action-acknowledged-by=sre-team

​

Chaos Engineering Correlation

• platform.chatcli.io/source=chaos-experiment
• platform.chatcli.io/chaos-experiment=<experiment-name>

​

SLA Integration

• Response time — max time from detection until the Issue is seen in Analyzing or Remediating
• Resolution time — max time from detection to resolution (an Issue that reaches Escalated is also checked against it)
• Business hours — optionally count only time inside business hours
• escalationPolicyRef / notificationPolicyRef — accepted by the CRD but not read by any controller: an SLA breach is recorded (status, metrics, audit) but does not trigger escalation or notifications by itself