Recipe: K8s Monitoring with AI

This recipe puts the K8s Watcher to work on an application myapp in the namespace production, in three setups:

Option	Runs	Good for
1. Local	`chatcli watch` on your machine, with your kubeconfig	one engineer, ad-hoc diagnosis
2. Team server	`chatcli server` with a watcher, deployed with Helm	a team sharing one context and one set of API keys
3. AIOps	an operator-managed `Instance`	alerts turned into Issues, AI analysis and remediation

Option 1: local monitoring

Check cluster access

kubectl config current-context
kubectl get deploy myapp -n production
kubectl auth can-i list pods -n production
kubectl auth can-i get pods/log -n production
kubectl auth can-i list events -n production

The watcher uses ~/.kube/config with its current context (or --kubeconfig); it does not read KUBECONFIG.

Start the watcher

chatcli watch --deployment myapp --namespace production

Starting K8s watcher: 1 targets (interval: 30s, window: 2h0m0s)
Initial data collected. K8s context will be injected into all prompts.
Type your questions about the deployments. K8s context is automatically included.
Use /watch to see current status.

Ask

> Is the deployment healthy?
> Why is pod myapp-7d9f-abcde restarting?
> What warning events happened in the last 30 minutes?
> Show the most recent error logs and what they have in common

Every prompt carries the deployment status, pods, events, recent error logs, HPA, node health and active alerts. /watch status prints the one-line state.

Tune it

Flag	Default	When to change
`--interval`	`30s`	`10s` while you actively debug; `60s` for a quiet environment
`--window`	`2h`	`30m` for a quick look; `6h` or more to compare with earlier
`--max-log-lines`	`100`	lower for chatty apps; higher for deep debugging

chatcli watch --deployment myapp --namespace production --interval 10s --window 6h --max-log-lines 300

Several workloads, other kinds, Prometheus metrics

# targets.yaml
interval: "15s"
window: "2h"
maxContextChars: 32000
targets:
  - deployment: frontend
    namespace: production
    metricsPort: 3000
    metricsFilter: ["http_*"]
  - deployment: backend
    namespace: production
    metricsPort: 9090
    metricsFilter: ["http_requests_*", "db_*", "cache_*"]
  - deployment: postgres
    kind: StatefulSet
    namespace: production
  - deployment: nightly-report
    kind: CronJob
    namespace: batch

chatcli watch --config targets.yaml

Targets with problems get full detail, healthy ones a single line, within maxContextChars. Prometheus scraping reads pod IPs over plain HTTP, so from a laptop it only works if your network routes to pod IPs; otherwise the other data still flows and the metrics section is simply absent.

One-shot for alert handlers

#!/usr/bin/env bash
# alert-handler.sh <deployment> <namespace>
set -euo pipefail
ANALYSIS=$(chatcli watch --deployment "$1" --namespace "$2" \
  -p "Analyze the current state of this deployment and identify the most likely root cause. Answer in Markdown.")

jq -n --arg text "*ChatCLI analysis of $2/$1*"$'\n\n'"$ANALYSIS" '{text: $text}' \
  | curl -sS -X POST -H 'Content-Type: application/json' --data @- "$SLACK_WEBHOOK_URL"

Each run collects once (up to 10 seconds) before asking, so the answer reflects the current state.

Option 2: team server with a watcher (Helm)

Install the server with a watcher

helm install chatcli oci://ghcr.io/diillson/charts/chatcli \
  --version 1.226.0 \
  --namespace monitoring --create-namespace \
  --set llm.provider=CLAUDEAI \
  --set secrets.anthropicApiKey="$ANTHROPIC_API_KEY" \
  --set server.token="$(openssl rand -hex 32)" \
  --set watcher.enabled=true \
  --set watcher.deployment=myapp \
  --set watcher.namespace=production \
  --set watcher.interval=15s

server.token is required: in a pod the server listens on every interface and refuses to start without a credential.
The server log reaches kubectl logs on its own: in a pod the server also writes it to stderr as JSON lines. The file copy in the pod rotates at 20 MB with 3 backups (logging.*), which fits the pod’s 200Mi data volume.
An empty watcher.namespace watches default. A namespace other than the release’s (production here, outside monitoring) makes the chart create a ClusterRole automatically. For several workloads, or a StatefulSet, DaemonSet, Job or CronJob (kind), use watcher.targets in a values file (see Helm watcher values).

Check it

kubectl -n monitoring rollout status deploy/chatcli
kubectl -n monitoring logs deploy/chatcli | grep -E 'listening|watcher active'
# 🚀 ChatCLI server listening on 0.0.0.0:50051
# K8s watcher active: 1 targets (interval: 15s)

Connect

export CHATCLI_REMOTE_TOKEN="$(kubectl -n monitoring get secret chatcli \
  -o jsonpath='{.data.CHATCLI_SERVER_TOKEN}' | base64 -d)"

kubectl -n monitoring port-forward svc/chatcli 50051:50051 &
CHATCLI_ALLOW_INSECURE=true chatcli connect localhost:50051 --token "$CHATCLI_REMOTE_TOKEN"

Connected to ChatCLI server (version: 1.226.0, provider: CLAUDEAI, model: claude-sonnet-5)
K8s watcher active: multi/1 targets (context injected into all prompts)

The server adds the watcher context to every prompt, so each teammate just asks. CHATCLI_ALLOW_INSECURE=true is needed because this server is plaintext; for anything beyond a port-forward turn on TLS and connect with --tls --ca-cert.

Distribute access with one token per person by switching to JWTs (security.jwtSecretRef), so each caller has its own identity and rate-limit bucket; see Server authentication.

Incident workflow

chatcli connect localhost:50051 --token "$CHATCLI_REMOTE_TOKEN"   # add CHATCLI_ALLOW_INSECURE=true for plaintext

> Summarize the current state of myapp for an incident channel
> What warning events occurred in the last 30 minutes?
> Show the most recent error logs
> What is the most likely root cause, and what should I do first?
> After the fix: are the pods back to normal? Compare with 20 minutes ago.

The comparison works within watcher.window (default 2 hours). /watch status in this session queries the server’s watcher (K8s Watcher (remote): …). For scripts against the server:

CHATCLI_ALLOW_INSECURE=true chatcli connect localhost:50051 --token "$CHATCLI_REMOTE_TOKEN" \
  -p "myapp is failing. Analyze and suggest a fix." --raw

Option 3: autonomous AIOps (operator)

The operator provisions the server from an Instance, reads the watcher’s alerts over StreamAlerts, and drives Anomaly → Issue → AIInsight → RemediationPlan. It dials the server over TLS with a credential, so the Instance needs both.

Install the operator

helm install chatcli-operator oci://ghcr.io/diillson/charts/chatcli-operator \
  --version 1.226.0 \
  -n chatcli-system --create-namespace

Create the Secrets

Follow Your first Instance for the three Secrets in the namespace chatcli: chatcli-api-keys (provider key), chatcli-server-token (key token) and chatcli-tls (tls.crt, tls.key, ca.crt, valid for chatcli.chatcli.svc.cluster.local).

Create the Instance with a watcher

apiVersion: platform.chatcli.io/v1alpha1
kind: Instance
metadata:
  name: chatcli
  namespace: chatcli
spec:
  provider: CLAUDEAI
  apiKeys:
    name: chatcli-api-keys
  server:
    port: 50051
    token:
      name: chatcli-server-token
      key: token
    tls:
      enabled: true
      secretName: chatcli-tls
  watcher:
    enabled: true
    interval: "15s"
    targets:
      - deployment: api-gateway
        namespace: production
        metricsPort: 9090
      - deployment: backend
        namespace: production
      - deployment: worker
        namespace: batch

kubectl apply -f instance.yaml
kubectl -n chatcli get instance chatcli

Without server.token (or another credential) the operator does not create the Deployment (AuthenticationConfigured=False); without TLS it cannot reach the server (ServerReachable=False). namespace is required on every target. The pipeline uses the first ready Instance in the cluster.

Follow the pipeline

kubectl get anomalies -A --watch
kubectl get issues -A
kubectl get aiinsights -A
kubectl get remediationplans -A
kubectl get runbooks -A
kubectl get postmortems -A

A crash-looping pod raises HighRestartCount (and OOMKilled when that is the cause); the operator records an Anomaly, correlates it into an Issue, asks the server for an analysis (AnalyzeIssue), and remediates from a matching Runbook, AI-suggested actions, or an agentic observe-decide-act loop (AgenticStep, up to spec.aiops.agenticMaxSteps, default 10). The full flow, approvals and safety limits are in AIOps Platform.

(Optional) Add a Runbook

apiVersion: platform.chatcli.io/v1alpha1
kind: Runbook
metadata:
  name: oom-standard-procedure
  namespace: production
spec:
  description: "Standard OOMKill recovery for production"
  trigger:
    signalType: oom_kill
    severity: critical
    resourceKind: Deployment
  steps:
    - name: Restart pods
      action: RestartDeployment
      description: "Restart to reclaim leaked memory"
    - name: Scale up
      action: ScaleDeployment
      description: "Add replicas for redundancy"
      params:
        replicas: "5"
  maxAttempts: 2

A manual Runbook that matches the Issue is used before AI-generated actions.

Troubleshooting

Symptom	Cause	Fix
`failed to create K8s watcher: …`	No usable kubeconfig or cluster unreachable	Check `kubectl get pods`; pass `--kubeconfig`
Context has no pods or events, log shows `forbidden`	Missing RBAC	Grant the watcher rules
Server pod crash-loops; `kubectl logs` shows `refusing to serve an unauthenticated API`	No credential	Set `server.token`
`tls: first record does not look like a TLS handshake` on connect	Plaintext server	`CHATCLI_ALLOW_INSECURE=true`, or enable TLS
`/watch status` after `chatcli connect` says no watcher	The server runs none: no `K8s watcher active` line at connect	Set `watcher.enabled=true` and a target on the server chart
No `## Application Metrics`	Pod IPs unreachable, wrong `metricsPort`/`metricsPath`, or filter matches nothing	Run the watcher in the cluster; check port and filter
Instance never gets a Deployment	`AuthenticationConfigured=False` or `TLSConfigured=False`	Add `server.token` and `tls.secretName`
No Anomalies appear	`ServerReachable=False`, or no alert raised yet	Check the Instance conditions; see K8s Operator troubleshooting

Get started

Documentation

Recipe: K8s Monitoring with AI

Option 1: local monitoring

Tune it

Several workloads, other kinds, Prometheus metrics

One-shot for alert handlers

Option 2: team server with a watcher (Helm)

Incident workflow

Option 3: autonomous AIOps (operator)

Troubleshooting

Checklist

​Option 1: local monitoring

​Tune it

​Several workloads, other kinds, Prometheus metrics

​One-shot for alert handlers

​Option 2: team server with a watcher (Helm)

​Incident workflow

​Option 3: autonomous AIOps (operator)

​Troubleshooting

​Checklist

Option 1: local monitoring

Tune it

Several workloads, other kinds, Prometheus metrics

One-shot for alert handlers

Option 2: team server with a watcher (Helm)

Incident workflow

Option 3: autonomous AIOps (operator)

Troubleshooting

Checklist