AIOps Platform (Deep-Dive)

​

Pipeline Overview

​

Internal Components

​

1. WatcherBridge (watcher_bridge.go)

hash = SHA256(alertType | deployment | namespace)

• No temporal component: A continuous problem (e.g., CrashLoopBackOff) generates only one Anomaly
• TTL: 2 hours — expired hashes are pruned automatically
• Invalidation: When an Issue reaches a terminal state (Resolved/Escalated), dedup entries for the affected resource are invalidated, allowing immediate recurrence detection
• Result: Avoids duplicates during an active problem; detects recurrence after resolution

​

2. AnomalyReconciler (anomaly_controller.go)

​

3. CorrelationEngine (correlation.go)

For each new anomaly:
  1. Generates incident_id = hash(resource_kind + resource_name + namespace + signal_type)
  2. Searches for existing Issue with the same incident_id
  3. If exists -> adds anomaly to Issue, recalculates risk score
  4. If not exists -> creates new Issue

risk_score >= 80 -> Critical
risk_score >= 60 -> High
risk_score >= 40 -> Medium
risk_score <  40 -> Low

​

4. IssueReconciler (issue_controller.go)

• Each retry triggers AI re-analysis with context from previous failures
• AI receives previous_failure_context with evidence from failed attempts
• The prompt instructs: “Do not repeat the same actions. Analyze why they failed and suggest a fundamentally different approach”
• Generates new auto-generated Runbook with different strategy (name includes attempt)

1. Existing manual Runbook (tiered match: SignalType+Severity+Kind -> Severity+Kind)
2. AI auto-generated Runbook (materialized as reusable CR)
3. Escalation (last resort)

​

5. AIInsightReconciler (aiinsight_controller.go)

• Deployment Status: replicas (desired/ready/updated/unavailable), conditions, container images + resources
• Pod Details (up to 5 pods, unhealthy first): phase, restart count, container states (Waiting/Terminated with reason + exit code)
• Recent Events (last 15): type, reason, message, count
• Revision History: Last 5 revisions (ReplicaSets) with image diff between revisions

​

6. RemediationReconciler (remediation_controller.go)

Pending -> Executing -> (executes actions sequentially) -> Verifying -> Completed | Failed

Pending -> Executing -> (agentic loop: AI decides -> executes -> observes -> repeat)
  -> Verifying -> Completed | Failed

Each reconcile = 1 step of the agentic loop:
  1. Refresh K8s context (KubernetesContextBuilder)
  2. Send history + context -> AgenticStep RPC
  3. AI responds: {reasoning, resolved, next_action}
  4. If resolved=true -> Verifying (+ annotations with PostMortem data)
  5. If next_action -> execute -> record observation -> requeue 5s
  6. If observation-only -> record -> requeue 10s
  Safety: max 10 steps, timeout 10 minutes

​

7. ServerClient (grpc_client.go)

​

Server and Operator Interaction

​

GetAlerts RPC

rpc GetAlerts(GetAlertsRequest) returns (GetAlertsResponse);

message AlertInfo {
  string alert_type = 1;    // HighRestartCount, OOMKilled, PodNotReady, DeploymentFailing
  string deployment = 2;
  string namespace = 3;
  string message = 4;
  string severity = 5;
  int64 timestamp = 6;
}

​

AnalyzeIssue RPC

rpc AnalyzeIssue(AnalyzeIssueRequest) returns (AnalyzeIssueResponse);

message SuggestedAction {
  string name = 1;
  string action = 2;
  string description = 3;
  map<string, string> params = 4;
}

message AnalyzeIssueResponse {
  string analysis = 1;
  float confidence = 2;
  repeated string recommendations = 3;
  string provider = 4;
  string model = 5;
  repeated SuggestedAction suggested_actions = 6;
}

1. Issue context (name, namespace, resource, severity, risk score, description)
2. List of available actions (ScaleDeployment, RestartDeployment, RollbackDeployment, PatchConfig)
3. Instructions to return structured JSON with analysis, confidence, recommendations, and actions fields

1. Removes markdown codeblocks (```json ... ```)
2. Parses JSON into analysisResult
3. Clamps confidence between 0.0 and 1.0
4. If parsing fails -> uses raw response as analysis with confidence 0.5

​

AgenticStep RPC

rpc AgenticStep(AgenticStepRequest) returns (AgenticStepResponse);

message AgenticStepRequest {
  string issue_name = 1;
  string namespace = 2;
  string resource_kind = 3;
  string resource_name = 4;
  string signal_type = 5;
  string severity = 6;
  string description = 7;
  int32 risk_score = 8;
  string provider = 9;
  string model = 10;
  string kubernetes_context = 11;   // refreshed at each step
  repeated AgenticHistoryEntry history = 12;
  int32 max_steps = 13;
  int32 current_step = 14;
}

message AgenticStepResponse {
  string reasoning = 1;              // AI reasoning (recorded in history)
  bool resolved = 2;                 // true = problem resolved
  SuggestedAction next_action = 3;   // null when resolved=true
  // Fields below only populated when resolved=true:
  string postmortem_summary = 4;
  string root_cause = 5;
  string impact = 6;
  repeated string lessons_learned = 7;
  repeated string prevention_actions = 8;
}

1. Role + Issue details: incident context (type, severity, resource)
2. Kubernetes context: real cluster state (refreshed at each step via KubernetesContextBuilder)
3. Tool definitions: 6 available mutating actions + “Observe” (no action, wait for next context)
4. Conversation history: each previous step formatted with reasoning -> action -> observation
5. Instructions: respond JSON, budget (step N of M), safety rules

​

PostMortem Generation

​

PostMortem CR

​

Auto-generated Runbook (Agentic)

• Name: agentic-{signal}-{severity}-{kind} (sanitized)
• Steps: only steps with successful actions
• Labels: auto-generated=true, source=agentic
• Uses CreateOrUpdate (reused for future incidents of the same type)

​

Operator Prometheus Metrics

​

Tests

​

Run Tests

cd operator
go test ./... -v

​

Ownership Diagram (Garbage Collection)

• Instance is the owner of all Kubernetes resources it creates (Deployment, Service, ConfigMap, SA, PVC)
• Issue is the owner of AIInsight, RemediationPlan, and PostMortem (cascade delete)
• Anomalies are independent (no owner) to preserve history

​

AIOps Deployment Checklist

​

Next Steps