AIOps Platform (Deep-Dive)

​

Platform v2 Components

​

REST API & Dashboard

• Authentication: header X-API-Key. Keys are read from the Secret chatcli-operator-secrets, key api-keys (fallback: ConfigMap chatcli-operator-config), in the operator namespace, as a YAML list of {key, role, description}. Roles are viewer < operator < admin; any other role string is denied. Changes are picked up within about 30 seconds; an api-keys entry that is not valid YAML keeps the last valid key set in force, and a Secret without the entry falls back to the ConfigMap. Create it yourself, or let the operator chart render it (apiKeys.create: true with apiKeys.entries). With no keys configured every /api/ call returns 401, unless CHATCLI_OPERATOR_DEV_MODE=true (admin without a key, development only).
• Rate limit: 30 requests per minute per client host without a valid API key, 600 per minute per valid key (429 with Retry-After).
• CORS: deny-all unless CHATCLI_CORS_ALLOWED_ORIGINS / CHATCLI_CORS_ORIGIN are set. TLS: set CHATCLI_AIOPS_TLS_CERT and CHATCLI_AIOPS_TLS_KEY to serve HTTPS (TLS 1.3).
• Local preview: make dash-preview in operator/ serves the dashboard over synthetic data, no cluster needed, at http://127.0.0.1:8085 with the API key preview.

​

Pipeline Overview

​

Internal Components

​

1. WatcherBridge (watcher_bridge.go)

hash = SHA256(alertType | deployment | namespace | resourceUID)

• No temporal component: A continuous problem (e.g., CrashLoopBackOff) generates only one Anomaly
• Resource UID: a deleted and recreated workload gets a new UID, so its alerts are not swallowed by the old entry
• TTL: 30 minutes by default (Instance spec.aiops.dedupTTLMinutes, 5–1440) — expired hashes are pruned automatically
• Invalidation: When an Issue is resolved, contained or escalated, dedup entries for the affected resource are invalidated, allowing immediate recurrence detection
• Result: Avoids duplicates during an active problem; detects recurrence after resolution

​

2. AnomalyReconciler (anomaly_controller.go)

​

3. CorrelationEngine (correlation.go)

For each new anomaly:
  1. Looks for a non-terminal Issue on the same resource (kind + name + namespace)
  2. If one exists -> attaches the anomaly (risk score recalculated, never lowered)
  3. Otherwise -> sums the weights of the uncorrelated anomalies on that resource
     from the last 10 minutes and creates a new Issue
  4. The incident ID INC-YYYYMMDD-NNN is a per-namespace, per-day sequence (label, not the Issue name)

signal oom_kill  -> Critical (always)
risk_score >= 80 -> Critical
risk_score >= 60 -> High
risk_score >= 30 -> Medium
risk_score <  30 -> Low

​

4. IssueReconciler (issue_controller.go)

• Each retry triggers AI re-analysis with context from previous failures
• AI receives previous_failure_context with evidence from failed attempts
• The prompt instructs: “Do not repeat the same actions. Analyze why they failed and suggest a fundamentally different approach”
• Generates a new auto-generated Runbook when the new analysis differs (the name carries a hash of the analysis)

1. Existing Runbook (tiered match: SignalType+Severity+Kind -> Severity+Kind)
2. AI auto-generated Runbook (materialized as reusable CR)
3. Agentic plan (AI decides each step)
4. Escalation after maxRemediationAttempts failed attempts

​

5. AIInsightReconciler (aiinsight_controller.go)

• Resource Status: replicas, conditions, containers, images + resources (each type has a dedicated context builder)
• StatefulSet: replicas, update strategy, partition, PodManagementPolicy, VolumeClaimTemplates
• DaemonSet: desired/current/ready/available/unavailable, nodeSelector, tolerations
• Job/CronJob: active/succeeded/failed, completions, parallelism, schedule, lastSuccessful
• HPA: min/max replicas, current/desired, target utilization, current metrics, maxed-out detection
• Pod Details (up to 5 pods, unhealthy first): phase, restart count, container states
• Recent Events (last 15): type, reason, message, count
• Revision History: Last 5 revisions (ReplicaSets) with image diff between revisions

• Stack Trace Extraction: detects and extracts stack traces from Java (Exception/Caused by), Go (panic/goroutine), Python (Traceback), Node.js (Error at)
• Error Pattern Detection: 24+ critical patterns categorized (crash, connectivity, dns, auth, storage, tls, database, cache, messaging)
• Structured Log Parsing: extracts error/warn entries from JSON logs (fields level, msg, error, timestamp, logger)
• Init Container Logs: analyzes init container logs (reveals startup failures)
• Sidecar Logs: analyzes sidecar logs (istio-proxy, envoy, datadog-agent, etc.)
• Critical Lines: extracts FATAL/PANIC lines with 3 lines of context before/after
• Temporal Window: fetches logs by temporal window (10min before the incident), not just tail

• CPU/Memory: usage trends 30min before → during → 15min after the incident
• Request/Error Rate: HTTP requests and 5xx per second
• Latency: P50, P95, P99 histogram percentiles
• HPA Metrics: current vs desired replicas, CPU target
• Network: receive/transmit bytes/s
• Trend Analysis: detects spikes, drops, sustained_high/low with % change calculation
• Enabled via: PROMETHEUS_URL env var on the operator

• Helm Releases: detects via Secrets type helm.sh/release.v1, status (deployed/failed/pending-upgrade), chart version, previous revision for rollback
• ArgoCD Applications: sync status (Synced/OutOfSync), health (Healthy/Degraded), conditions, last sync result
• Flux Kustomizations: ready status, source ref, conditions, last applied

• Git Correlation: finds commits in the 30min before the incident
• Suspected Commit: identifies the most likely commit (score by temporal proximity + volume of changes)
• Code Extraction: extracts code snippets referenced in stack traces (file path + line number → source code)
• Config Analysis: reads Dockerfile, values.yaml, Chart.yaml for deploy context
• Credentials: Secret keys token, username + password, or ssh-key + known_hosts; HTTPS credentials go through a GIT_ASKPASS helper and are never written to .git/config; SSH verifies host keys against known_hosts (spec.sshHostKeyPolicy: acceptNew trusts the first key when the Secret has none). See Source repositories

• Dependency Graph: discovers dependencies via Services + EndpointSlices
• Temporal Correlation: finds active issues in the same namespace and cross-namespace within a 15-20min window
• Cascade Chain: orders services by detection time (first = root cause)
• Root Cause Service: identifies the service that originated the cascade

• PDB Check: verifies if the action would violate PodDisruptionBudgets
• Quota Check: verifies ResourceQuotas (>90% used = warning)
• Node Capacity: counts pods on node for cordon/drain actions
• Affected Services: discovers which Services would be impacted
• Risk Level: classifies as low/medium/high/critical

​

6. RemediationReconciler (remediation_controller.go)

Pending -> Snapshot -> Executing -> (checkpoint + action, for each action)
  -> Verifying (90s timeout) -> Completed
  -> If action fails -> Automatic rollback -> RolledBack
  -> If verification fails -> Automatic rollback -> RolledBack
  -> If rollback fails -> Failed

Pending -> Executing -> (agentic loop: AI decides -> executes -> observes -> repeat)
  -> Verifying -> Completed | Failed

Each reconcile = 1 step of the agentic loop:
  1. Refresh K8s context (KubernetesContextBuilder)
  2. Send history + context -> AgenticStep RPC
  3. AI responds: {reasoning, resolved, next_action}
  4. If resolved=true -> Verifying (+ annotations with PostMortem data)
  5. If next_action -> execute -> record observation -> requeue 5s
  6. If observation-only -> record -> requeue 10s
  7. If next_action diverges from the AIInsight without a divergence_reason
     -> recorded as rejected, not executed, the AI is asked again
  Safety: max steps (Instance aiops.agenticMaxSteps, default 10), timeout 10 minutes,
  convergence detector (repeated observations, A-B-A-B oscillation, 5 failures in a row,
  more than 8 minutes elapsed) fails the plan early

​

7. ServerClient (grpc_client.go)

​

Server and Operator Interaction

​

StreamAlerts and GetAlerts RPCs

rpc GetAlerts(GetAlertsRequest) returns (GetAlertsResponse);
rpc StreamAlerts(StreamAlertsRequest) returns (stream StreamAlertsResponse);

message WatcherAlert {
  string type = 1;            // HighRestartCount, OOMKilled, PodNotReady, DeploymentFailing, ...
  string severity = 2;        // info, warning, critical
  string message = 3;
  string object = 4;          // e.g. pod name
  string namespace = 5;
  string deployment = 6;
  int64 timestamp_unix = 7;
}

message GetAlertsRequest {
  string namespace = 1;       // optional filter
  string deployment = 2;      // optional filter
}

​

AnalyzeIssue RPC

rpc AnalyzeIssue(AnalyzeIssueRequest) returns (AnalyzeIssueResponse);

message SuggestedAction {
  string name = 1;
  string action = 2;
  string description = 3;
  map<string, string> params = 4;
}

message AnalyzeIssueResponse {
  string analysis = 1;
  float confidence = 2;       // 0.0 to 1.0
  repeated string recommendations = 3;
  string model = 4;
  string provider = 5;
  repeated SuggestedAction suggested_actions = 6;
  TokenUsage usage = 7;       // feeds the cost ledger
}

1. Issue context (name, namespace, resource, severity, risk score, description)
2. The action catalog (the 54 action types, with their parameters and resource-kind rules)
3. Instructions to return structured JSON with analysis, confidence, recommendations, and actions fields

1. Removes markdown codeblocks (```json ... ```)
2. Parses JSON into analysisResult
3. Clamps confidence between 0.0 and 1.0
4. If parsing fails -> uses raw response as analysis with confidence 0.5

​

AgenticStep RPC

rpc AgenticStep(AgenticStepRequest) returns (AgenticStepResponse);

message AgenticStepRequest {
  string issue_name = 1;
  string namespace = 2;
  string resource_kind = 3;
  string resource_name = 4;
  string signal_type = 5;
  string severity = 6;
  string description = 7;
  int32 risk_score = 8;
  string provider = 9;
  string model = 10;
  string kubernetes_context = 11;   // refreshed at each step
  repeated AgenticHistoryEntry history = 12;
  int32 max_steps = 13;
  int32 current_step = 14;
  // Prior AIInsight guidance, so the loop cannot silently contradict its own analysis
  string insight_analysis = 15;
  float insight_confidence = 16;
  repeated string insight_recommendations = 17;
  repeated SuggestedAction insight_suggested_actions = 18;
}

message AgenticStepResponse {
  string reasoning = 1;              // AI reasoning (recorded in history)
  bool resolved = 2;                 // true = problem resolved
  SuggestedAction next_action = 3;   // null when resolved=true
  // Fields below only populated when resolved=true:
  string postmortem_summary = 4;
  string root_cause = 5;
  string impact = 6;
  repeated string lessons_learned = 7;
  repeated string prevention_actions = 8;
  bool diverges_from_insight = 9;     // next_action contradicts the AIInsight
  string divergence_reason = 10;      // justification when diverging on purpose
  TokenUsage usage = 11;
}

1. Role + Issue details: incident context (type, severity, resource)
2. Kubernetes context: real cluster state (refreshed at each step via KubernetesContextBuilder)
3. Tool definitions: the action catalog + “Observe” (no action, wait for next context)
4. Conversation history: each previous step formatted with reasoning -> action -> observation
5. Instructions: respond JSON, budget (step N of M), safety rules

​

PostMortem Generation

​

PostMortem CR

• Trending: detection of recurring incidents (count in the last 30 days, related PostMortems)
• Cascade Chain: cascade failure chain if there are correlated cross-service issues
• Git Correlation: suspected commit (SHA, author, changed files, confidence)
• GitOps Context: Helm/ArgoCD/Flux state at the time of the incident

​

Auto-generated Runbook (Agentic)

• Name: agentic-{signal}-{severity}-{kind} (sanitized)
• Steps: only steps with successful actions
• Labels: auto-generated=true, source=agentic
• Uses CreateOrUpdate (reused for future incidents of the same type)

​

Operator Prometheus Metrics

​

Tests

​

Run Tests

cd operator
go test ./... -v          # unit tests over the fake client
make test-integration    # envtest: real API server, CRDs, every controller wired

​

Ownership Diagram (Garbage Collection)

• Instance is the owner of the namespaced resources it creates (Deployment, Service, ConfigMaps, SA, PVC, watcher Role/RoleBinding); a cross-namespace watcher ClusterRoleBinding is removed by the Instance finalizer instead
• Issue is the owner of AIInsight, RemediationPlan, and PostMortem (cascade delete)
• Anomalies are independent (no owner) to preserve history

​

AIOps Deployment Checklist

helm install chatcli-operator \
  oci://ghcr.io/diillson/charts/chatcli-operator \
  --version 1.214.0 \
  --namespace chatcli-system --create-namespace

kubectl create namespace chatcli
kubectl -n chatcli create secret generic chatcli-api-keys --from-literal=ANTHROPIC_API_KEY=<your-key>
kubectl -n chatcli create secret generic chatcli-server-token --from-literal=token="$(openssl rand -hex 32)"
kubectl -n chatcli create secret tls chatcli-tls --cert=tls.crt --key=tls.key

apiVersion: platform.chatcli.io/v1alpha1
kind: Instance
metadata:
  name: chatcli
  namespace: chatcli
spec:
  provider: CLAUDEAI
  apiKeys:
    name: chatcli-api-keys
  server:
    tls:
      enabled: true
      secretName: chatcli-tls
    token:
      name: chatcli-server-token
      key: token
  watcher:
    enabled: true
    targets:
      - name: my-app
        kind: Deployment
        namespace: production

​

Next Steps