AIOps Platform (Deep-Dive)

​

Platform v2 Components

​

REST API & Dashboard

​

Pipeline Overview

​

Internal Components

​

1. WatcherBridge (watcher_bridge.go)

hash = SHA256(alertType | deployment | namespace)

• No temporal component: A continuous problem (e.g., CrashLoopBackOff) generates only one Anomaly
• TTL: 2 hours — expired hashes are pruned automatically
• Invalidation: When an Issue reaches a terminal state (Resolved/Escalated), dedup entries for the affected resource are invalidated, allowing immediate recurrence detection
• Result: Avoids duplicates during an active problem; detects recurrence after resolution

​

2. AnomalyReconciler (anomaly_controller.go)

​

3. CorrelationEngine (correlation.go)

For each new anomaly:
  1. Generates incident_id = hash(resource_kind + resource_name + namespace + signal_type)
  2. Searches for existing Issue with the same incident_id
  3. If exists -> adds anomaly to Issue, recalculates risk score
  4. If not exists -> creates new Issue

risk_score >= 80 -> Critical
risk_score >= 60 -> High
risk_score >= 40 -> Medium
risk_score <  40 -> Low

​

4. IssueReconciler (issue_controller.go)

• Each retry triggers AI re-analysis with context from previous failures
• AI receives previous_failure_context with evidence from failed attempts
• The prompt instructs: “Do not repeat the same actions. Analyze why they failed and suggest a fundamentally different approach”
• Generates new auto-generated Runbook with different strategy (name includes attempt)

1. Existing manual Runbook (tiered match: SignalType+Severity+Kind -> Severity+Kind)
2. AI auto-generated Runbook (materialized as reusable CR)
3. Escalation (last resort)

​

5. AIInsightReconciler (aiinsight_controller.go)

• Resource Status: replicas, conditions, containers, images + resources (each type has a dedicated context builder)
• StatefulSet: replicas, update strategy, partition, PodManagementPolicy, VolumeClaimTemplates
• DaemonSet: desired/current/ready/available/unavailable, nodeSelector, tolerations
• Job/CronJob: active/succeeded/failed, completions, parallelism, schedule, lastSuccessful
• HPA: min/max replicas, current/desired, target utilization, current metrics, maxed-out detection
• Pod Details (up to 5 pods, unhealthy first): phase, restart count, container states
• Recent Events (last 15): type, reason, message, count
• Revision History: Last 5 revisions (ReplicaSets) with image diff between revisions

• Stack Trace Extraction: detects and extracts stack traces from Java (Exception/Caused by), Go (panic/goroutine), Python (Traceback), Node.js (Error at)
• Error Pattern Detection: 24+ critical patterns categorized (crash, connectivity, dns, auth, storage, tls, database, cache, messaging)
• Structured Log Parsing: extracts error/warn entries from JSON logs (fields level, msg, error, timestamp, logger)
• Init Container Logs: analyzes init container logs (reveals startup failures)
• Sidecar Logs: analyzes sidecar logs (istio-proxy, envoy, datadog-agent, etc.)
• Critical Lines: extracts FATAL/PANIC lines with 3 lines of context before/after
• Temporal Window: fetches logs by temporal window (10min before the incident), not just tail

• CPU/Memory: usage trends 30min before → during → 15min after the incident
• Request/Error Rate: HTTP requests and 5xx per second
• Latency: P50, P95, P99 histogram percentiles
• HPA Metrics: current vs desired replicas, CPU target
• Network: receive/transmit bytes/s
• Trend Analysis: detects spikes, drops, sustained_high/low with % change calculation
• Enabled via: PROMETHEUS_URL env var on the operator

• Helm Releases: detects via Secrets type helm.sh/release.v1, status (deployed/failed/pending-upgrade), chart version, previous revision for rollback
• ArgoCD Applications: sync status (Synced/OutOfSync), health (Healthy/Degraded), conditions, last sync result
• Flux Kustomizations: ready status, source ref, conditions, last applied

• Git Correlation: finds commits in the 30min before the incident
• Suspected Commit: identifies the most likely commit (score by temporal proximity + volume of changes)
• Code Extraction: extracts code snippets referenced in stack traces (file path + line number → source code)
• Config Analysis: reads Dockerfile, values.yaml, Chart.yaml for deploy context

• Dependency Graph: discovers dependencies via Services + EndpointSlices
• Temporal Correlation: finds active issues in the same namespace and cross-namespace within a 15-20min window
• Cascade Chain: orders services by detection time (first = root cause)
• Root Cause Service: identifies the service that originated the cascade

• PDB Check: verifies if the action would violate PodDisruptionBudgets
• Quota Check: verifies ResourceQuotas (>90% used = warning)
• Node Capacity: counts pods on node for cordon/drain actions
• Affected Services: discovers which Services would be impacted
• Risk Level: classifies as low/medium/high/critical

​

6. RemediationReconciler (remediation_controller.go)

Pending -> Snapshot -> Executing -> (checkpoint + action, for each action)
  -> Verifying (90s timeout) -> Completed
  -> If action fails -> Automatic rollback -> RolledBack
  -> If verification fails -> Automatic rollback -> RolledBack
  -> If rollback fails -> Failed

Pending -> Executing -> (agentic loop: AI decides -> executes -> observes -> repeat)
  -> Verifying -> Completed | Failed

Each reconcile = 1 step of the agentic loop:
  1. Refresh K8s context (KubernetesContextBuilder)
  2. Send history + context -> AgenticStep RPC
  3. AI responds: {reasoning, resolved, next_action}
  4. If resolved=true -> Verifying (+ annotations with PostMortem data)
  5. If next_action -> execute -> record observation -> requeue 5s
  6. If observation-only -> record -> requeue 10s
  Safety: max 10 steps, timeout 10 minutes

​

7. ServerClient (grpc_client.go)

​

Server and Operator Interaction

​

GetAlerts RPC

rpc GetAlerts(GetAlertsRequest) returns (GetAlertsResponse);

message AlertInfo {
  string alert_type = 1;    // HighRestartCount, OOMKilled, PodNotReady, DeploymentFailing
  string deployment = 2;
  string namespace = 3;
  string message = 4;
  string severity = 5;
  int64 timestamp = 6;
}

​

AnalyzeIssue RPC

rpc AnalyzeIssue(AnalyzeIssueRequest) returns (AnalyzeIssueResponse);

message SuggestedAction {
  string name = 1;
  string action = 2;
  string description = 3;
  map<string, string> params = 4;
}

message AnalyzeIssueResponse {
  string analysis = 1;
  float confidence = 2;
  repeated string recommendations = 3;
  string provider = 4;
  string model = 5;
  repeated SuggestedAction suggested_actions = 6;
}

1. Issue context (name, namespace, resource, severity, risk score, description)
2. List of 19 available actions organized by category (Workload, GitOps, Autoscaling, Infra, Storage, Security, Networking, Advanced)
3. Instructions to return structured JSON with analysis, confidence, recommendations, and actions fields

1. Removes markdown codeblocks (```json ... ```)
2. Parses JSON into analysisResult
3. Clamps confidence between 0.0 and 1.0
4. If parsing fails -> uses raw response as analysis with confidence 0.5

​

AgenticStep RPC

rpc AgenticStep(AgenticStepRequest) returns (AgenticStepResponse);

message AgenticStepRequest {
  string issue_name = 1;
  string namespace = 2;
  string resource_kind = 3;
  string resource_name = 4;
  string signal_type = 5;
  string severity = 6;
  string description = 7;
  int32 risk_score = 8;
  string provider = 9;
  string model = 10;
  string kubernetes_context = 11;   // refreshed at each step
  repeated AgenticHistoryEntry history = 12;
  int32 max_steps = 13;
  int32 current_step = 14;
}

message AgenticStepResponse {
  string reasoning = 1;              // AI reasoning (recorded in history)
  bool resolved = 2;                 // true = problem resolved
  SuggestedAction next_action = 3;   // null when resolved=true
  // Fields below only populated when resolved=true:
  string postmortem_summary = 4;
  string root_cause = 5;
  string impact = 6;
  repeated string lessons_learned = 7;
  repeated string prevention_actions = 8;
}

1. Role + Issue details: incident context (type, severity, resource)
2. Kubernetes context: real cluster state (refreshed at each step via KubernetesContextBuilder)
3. Tool definitions: 18 available mutating actions + “Observe” (no action, wait for next context)
4. Conversation history: each previous step formatted with reasoning -> action -> observation
5. Instructions: respond JSON, budget (step N of M), safety rules

​

PostMortem Generation

​

PostMortem CR

• Trending: detection of recurring incidents (count in the last 30 days, related PostMortems)
• Cascade Chain: cascade failure chain if there are correlated cross-service issues
• Git Correlation: suspected commit (SHA, author, changed files, confidence)
• GitOps Context: Helm/ArgoCD/Flux state at the time of the incident

​

Auto-generated Runbook (Agentic)

• Name: agentic-{signal}-{severity}-{kind} (sanitized)
• Steps: only steps with successful actions
• Labels: auto-generated=true, source=agentic
• Uses CreateOrUpdate (reused for future incidents of the same type)

​

Operator Prometheus Metrics

​

Tests

​

Run Tests

cd operator
go test ./... -v

​

Ownership Diagram (Garbage Collection)

• Instance is the owner of all Kubernetes resources it creates (Deployment, Service, ConfigMap, SA, PVC)
• Issue is the owner of AIInsight, RemediationPlan, and PostMortem (cascade delete)
• Anomalies are independent (no owner) to preserve history

​

AIOps Deployment Checklist

helm install chatcli-operator \
  oci://ghcr.io/diillson/charts/chatcli-operator \
  --namespace chatcli-system --create-namespace

​

Next Steps