Recipe: K8s Monitoring with AI

# Verify connectivity
kubectl get pods -n production

# Verify permissions
kubectl auth can-i get pods -n production
kubectl auth can-i get pods/log -n production
kubectl auth can-i list events -n production

chatcli watch --deployment myapp --namespace production

K8s Watcher starting...
  Deployment: myapp
  Namespace:  production
  Interval:   30s
  Window:     2h

Collecting initial data...
Initial data collected. Starting interactive mode.
[watch] chatcli>

[watch] chatcli> Is the deployment healthy?

Based on the collected Kubernetes data:
- The myapp deployment has 3/3 replicas available
- All pods are in Running and Ready state
- There are no active alerts
- Average CPU at 35%, memory at 120Mi
The deployment is healthy and operating normally.

[watch] chatcli> /watch status

K8s Watcher Active
  Deployment:  myapp
  Namespace:   production
  Snapshots:   5
  Pods:        3
  Alerts:      0

[watch] chatcli> Why is the pod myapp-abc12 restarting?

Analyzing data for pod myapp-abc12:
- The pod had 5 restarts in the last hour
- Reason for last restart: OOMKilled
- Container was using 490Mi out of 512Mi limit
- Logs show: "java.lang.OutOfMemoryError: Java heap space"

Diagnosis: The container is exceeding the memory limit.
Recommendations:
1. Increase resources.limits.memory to 1Gi
2. Adjust the JVM: -Xmx384m to fit within the limit
3. Investigate possible memory leak in previous logs

# Each dev configures
export CHATCLI_REMOTE_ADDR=chatcli.monitoring.svc:50051
export CHATCLI_REMOTE_TOKEN=team-token

# Via port-forward (development)
kubectl port-forward -n monitoring svc/chatcli 50051:50051
chatcli connect localhost:50051 --token team-token

> What is happening with the deployment?

[The server automatically injects K8s Watcher data]

chatcli connect prod-chatcli:50051 --token ops-token

> Summarize the current state of the deployment for a post-mortem

> What Warning events occurred in the last 30 minutes?
> Show the most recent error logs
> What changed since the last deploy?

> Based on the data, what is the most likely root cause and what
  should I do to resolve it?

> After applying the fix, are the pods returning to normal?
> Compare the current state with 10 minutes ago

# Install CRDs
kubectl apply -f operator/config/crd/bases/

# Install RBAC and Manager
kubectl apply -f operator/config/rbac/role.yaml
kubectl apply -f operator/config/manager/manager.yaml

apiVersion: platform.chatcli.io/v1alpha1
kind: Instance
metadata:
  name: chatcli-aiops
  namespace: monitoring
spec:
  provider: CLAUDEAI
  apiKeys:
    name: chatcli-api-keys
  server:
    port: 50051
  watcher:
    enabled: true
    interval: "15s"
    targets:
      - deployment: api-gateway
        namespace: production
        metricsPort: 9090
      - deployment: backend
        namespace: production
        metricsPort: 9090
      - deployment: worker
        namespace: batch

# Check detected anomalies
kubectl get anomalies -A --watch

# Check created issues
kubectl get issues -A --watch

# Check AI analyses
kubectl get aiinsights -A

# Check remediations
kubectl get remediationplans -A

# Check runbooks (manual and auto-generated)
kubectl get runbooks -A

# Check post-mortems (generated after agentic resolution)
kubectl get postmortems -A

1. WatcherBridge detects HighRestartCount -> creates Anomaly
2. AnomalyReconciler correlates -> creates Issue (risk: 20, severity: Low, signalType: pod_restart)
3. If OOMKilled also -> Issue updated (risk: 50, severity: Medium)
4. IssueReconciler creates AIInsight
5. AIInsightReconciler collects K8s context (pods, events, revisions)
   -> calls LLM with enriched context -> returns: "restart + scale to 4"
6. IssueReconciler looks up manual Runbook (tiered matching)
   -> if not found, generates auto Runbook CR from AI (reusable)
   -> if no Runbook and no AI actions -> Agential mode
   -> creates RemediationPlan with actions
7. RemediationReconciler executes:
   -> Normal mode: executes plan actions (restart, scale, etc.)
   -> Agential mode: observe-decide-act loop (AI decides each step)
     - Each reconcile = 1 step (max 10 steps, timeout 10min)
     - AI observes K8s state -> decides action -> executes -> next step
     - Actions: Scale, Restart, Rollback, PatchConfig, AdjustResources, DeletePod
8. Issue -> Resolved (dedup invalidated for recurrence detection)
   -> If agential mode: PostMortem CR auto-generated (timeline, root cause, lessons)
   -> Runbook auto-generated from successful agential steps
   If failed -> re-analysis with failure context -> different strategy

apiVersion: platform.chatcli.io/v1alpha1
kind: Runbook
metadata:
  name: oom-standard-procedure
  namespace: production
spec:
  description: "Standard OOMKill recovery for production"
  trigger:
    signalType: oom_kill
    severity: critical
    resourceKind: Deployment
  steps:
    - name: Restart pods
      action: RestartDeployment
      description: "Restart to reclaim leaked memory"
    - name: Scale up
      action: ScaleDeployment
      description: "Add replicas for redundancy"
      params:
        replicas: "5"
  maxAttempts: 2