AIOps: Complete Production Setup

This runbook takes an empty cluster to a working ChatCLI AIOps platform. Every manifest below validates against the current CRDs. Run the steps in order. Most of them end with a check to run before you move on. At the end you will have:

the operator in chatcli-system, with its REST API, web dashboard, metrics and an optional NetworkPolicy;
a ChatCLI server (Instance) in chatcli, with a credential, TLS, explicit resources and a watcher on your production workloads;
notification, escalation, SLO, SLA and approval policies;
Grafana dashboards and Prometheus scraping;
a chaos drill that shows the pipeline detecting a failure.

Prerequisites

A Kubernetes cluster, and cluster-admin rights for the install (the chart creates CRDs, ClusterRoles and ClusterRoleBindings).
kubectl, Helm 3.8 or later (for OCI charts; Helm 4 included) and openssl.
At least one LLM provider key (the examples use Anthropic and OpenAI).
For observability: the Prometheus Operator (for ServiceMonitor) and Grafana with the dashboard sidecar. Both are optional.
The chatcli CLI on your workstation, for the end-to-end check in Verify.

Where each resource lives

The operator looks resources up in specific namespaces. Putting one in the wrong namespace does not fail. It is silently ignored.

Resource	Namespace	Why
Operator, and the dashboard key Secret `chatcli-operator-secrets`	The operator’s namespace (`chatcli-system`)	The operator reads that fixed-name Secret from its own namespace
`Instance` and every Secret it references	Its own namespace (`chatcli`)	Secret references on a pod are always same-namespace
Anomaly, Issue, AIInsight, RemediationPlan, PostMortem	The affected workload’s namespace (`production`)	The operator creates them there
`ApprovalPolicy`	The workload’s namespace	Only policies in the plan’s namespace are evaluated
`ServiceLevelObjective`	The workload’s namespace	SLO lookups stay in the SLO’s namespace
`SourceRepository` and its git Secret	The workload’s namespace	The repository is matched in the resource’s namespace, and the Secret in the repository’s namespace
`ChaosExperiment`	The target’s namespace	Issue correlation only searches the target’s namespace
`NotificationPolicy`, `EscalationPolicy` and their Secrets	Any (this page uses `chatcli-system`)	Every enabled policy in the cluster is evaluated; Secrets are read from the policy’s namespace
`IncidentSLA`	Any	One in the Issue’s namespace wins; otherwise one from any namespace is used

Only one Instance per cluster drives AIOps. The operator’s alert bridge connects to the first ready Instance it lists across all namespaces, and the AI analysis and remediation use that same connection. If several Instances are ready, which one it picks is not defined. Run a single Instance with a watcher per cluster.

1. Install the operator

Write a values file. Every key below exists in the chart’s schema, and the schema rejects unknown keys.

# values-operator.yaml
replicaCount: 2            # leader election is on; the REST API and dashboard serve on every replica
leaderElect: true

# Prometheus the operator queries during incident analysis (optional, see step 10)
prometheusUrl: "http://prometheus-operated.monitoring.svc:9090"

serviceMonitor:
  enabled: true            # requires the Prometheus Operator CRDs
  labels:
    release: kube-prometheus-stack   # whatever label your Prometheus selects ServiceMonitors by

networkPolicy:
  enabled: true
  apiIngressFrom:          # who may reach the REST API and dashboard (port 8090)
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: ingress-nginx
  metricsIngressFrom:      # who may scrape the metrics port (8080)
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: monitoring
  egress: restricted       # DNS, 443, the API server, the Instances' gRPC port, Prometheus
  kubernetesApiPort: 6443  # use 443 if your API server listens there
  egressExtraPorts:
    - port: 587            # SMTP for email notifications (step 5)
      protocol: TCP

# The root filesystem is read-only; the chart mounts a writable /tmp
# emptyDir, where SourceRepository clones live (step 4). Raise the limit
# for large repositories.
tmpVolume:
  sizeLimit: 1Gi

helm install chatcli-operator oci://ghcr.io/diillson/charts/chatcli-operator \
  --version 1.226.0 \
  --namespace chatcli-system --create-namespace \
  -f values-operator.yaml

The chart installs the 17 CRDs, the operator Deployment, its Service (chatcli-operator, ports metrics 8080, health 8081, api 8090), cluster RBAC, the pre-provisioned chatcli-watcher ClusterRole, and a pre-install/pre-upgrade hook that re-applies the CRDs on every helm upgrade. It pins the server image that Instances use to the same release, unless an Instance sets spec.image.tag. Check:

kubectl get crd -o name | grep -c platform.chatcli.io     # 17
kubectl -n chatcli-system rollout status deploy/chatcli-operator

The operator chart has no PodDisruptionBudget template. With replicaCount: 2, create one yourself if node drains must keep a replica up (selector app.kubernetes.io/name: chatcli-operator).

2. Create the Instance Secrets

All of these go in the Instance’s namespace.

kubectl create namespace chatcli

# LLM provider keys, loaded into the server with envFrom
kubectl -n chatcli create secret generic chatcli-llm-keys \
  --from-literal=ANTHROPIC_API_KEY='<anthropic-key>' \
  --from-literal=OPENAI_API_KEY='<openai-key>'

# Shared server token. It is mandatory: in Kubernetes the server listens on
# every interface and refuses to start without a credential.
kubectl -n chatcli create secret generic chatcli-prod-token \
  --from-literal=token="$(openssl rand -hex 32)"

# Key for encryption at rest of what the server writes under ~/.chatcli
kubectl -n chatcli create secret generic chatcli-encryption-key \
  --from-literal=key="$(openssl rand -hex 32)"

The operator watches every Secret an Instance references. When you rotate one, the pods roll automatically, because a hash of the referenced keys is stamped on the pod template.

TLS certificate

The operator always dials the server over TLS 1.3. It has no plaintext mode, so an Instance without TLS runs but the operator cannot reach it (ServerReachable=False), and alerts, AI analysis and remediation stop there. The operator dials <instance>.<namespace>.svc.cluster.local, here chatcli-prod.chatcli.svc.cluster.local. The certificate must be valid for that name. If it is not signed by a public CA, the Secret must also carry the CA as ca.crt, which the operator uses as its trust root for this Instance.

cert-manager (recommended)
openssl (self-signed)

An internal CA issuer writes tls.crt, tls.key and ca.crt into the Secret, and renews the certificate. Each renewal rolls the Instance pods.

# Bootstrap a private CA (skip if you already have a CA ClusterIssuer)
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: selfsigned-bootstrap
spec:
  selfSigned: {}
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: chatcli-internal-ca
  namespace: cert-manager        # the cluster resource namespace of cert-manager
spec:
  isCA: true
  commonName: chatcli-internal-ca
  secretName: chatcli-internal-ca
  privateKey:
    algorithm: ECDSA
    size: 256
  issuerRef:
    name: selfsigned-bootstrap
    kind: ClusterIssuer
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: chatcli-internal-ca
spec:
  ca:
    secretName: chatcli-internal-ca
---
# The server certificate
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: chatcli-prod-tls
  namespace: chatcli
spec:
  secretName: chatcli-prod-tls
  issuerRef:
    name: chatcli-internal-ca
    kind: ClusterIssuer
  dnsNames:
    - chatcli-prod.chatcli.svc.cluster.local   # the name the operator dials
    - chatcli-prod.chatcli.svc
    - chatcli-prod
    - localhost                                # for the port-forward check in step 13
  duration: 2160h      # 90 days
  renewBefore: 360h    # 15 days

kubectl -n chatcli wait certificate/chatcli-prod-tls --for=condition=Ready --timeout=2m
kubectl -n chatcli get secret chatcli-prod-tls -o jsonpath='{.data}' | grep -o '"[a-z.]*":' # ca.crt, tls.crt, tls.key

A self-signed certificate is its own CA, so the same file goes in as ca.crt. You have to rotate it yourself.

openssl req -x509 -newkey rsa:4096 -sha256 -days 365 -nodes \
  -keyout tls.key -out tls.crt \
  -subj "/CN=chatcli-prod.chatcli.svc.cluster.local" \
  -addext "subjectAltName=DNS:chatcli-prod.chatcli.svc.cluster.local,DNS:chatcli-prod.chatcli.svc,DNS:chatcli-prod,DNS:localhost"

openssl x509 -in tls.crt -noout -ext subjectAltName   # check the SANs

kubectl -n chatcli create secret generic chatcli-prod-tls \
  --from-file=tls.crt=tls.crt \
  --from-file=tls.key=tls.key \
  --from-file=ca.crt=tls.crt

Issuer	`ca.crt` in the Secret	Notes
cert-manager with an internal CA issuer	Written by cert-manager	Put the in-cluster names in `dnsNames`
Private CA (for example AWS Private CA)	Add the CA bundle as `ca.crt` yourself	Issue the certificate for the in-cluster names
Self-signed	`ca.crt` = the certificate itself	Rotate it yourself
Public CA (for example Let’s Encrypt)	Not needed: the operator falls back to the system CAs	Only works if the certificate covers `<instance>.<namespace>.svc.cluster.local`, which public CAs do not issue. In practice, use a private CA for the in-cluster hop

There is no Instance field for the address the operator dials. It is always the Instance Service name above, on spec.server.port. Operator-wide overrides exist for the trust root and a client certificate (chart security.grpcTLS.*), but a ca.crt in the Instance’s own Secret is simpler and wins over them.

3. Create the production Instance

apiVersion: platform.chatcli.io/v1alpha1
kind: Instance
metadata:
  name: chatcli-prod
  namespace: chatcli
spec:
  replicas: 2
  provider: CLAUDEAI
  model: claude-sonnet-5
  # No image.tag: the server image is pinned to the operator's release.

  apiKeys:
    name: chatcli-llm-keys          # every key of this Secret becomes an env var

  fallback:
    enabled: true
    providers:                      # the operator puts spec.provider first for you
      - name: OPENAI
        model: gpt-6-sol

  server:
    port: 50051
    metricsPort: 9090               # also serves /healthz for the pod probes; 0 still means 9090
    token:
      name: chatcli-prod-token
      key: token
    tls:
      enabled: true
      secretName: chatcli-prod-tls  # tls.crt, tls.key, ca.crt
    security:
      rateLimitRps: 20              # per authenticated subject; every shared-token caller shares one bucket
      rateLimitBurst: 40

  watcher:
    enabled: true
    interval: "30s"
    window: "2h"
    maxLogLines: 100
    targets:
      - name: api-gateway
        kind: Deployment
        namespace: production
        metricsPort: 8080
        metricsPath: /metrics
      - name: payment-service
        kind: Deployment
        namespace: production
      - name: orders-db
        kind: StatefulSet
        namespace: production

  aiops:
    maxRemediationAttempts: 3       # default 5
    dedupTTLMinutes: 30

  # The operator applies NO default resources: set them explicitly.
  resources:
    requests:
      cpu: 250m
      memory: 512Mi
    limits:
      cpu: "1"
      memory: 1Gi

  # With replicas > 1 keep persistence off (see the note below).
  persistence:
    enabled: false

  features:
    encryptionKeyRef:
      name: chatcli-encryption-key
      key: key
    # caBundleSecretName: corporate-ca   # Secret with ca.crt, for a TLS-inspecting egress proxy

  scheduling:
    affinity:
      podAntiAffinity:
        preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 100
            podAffinityTerm:
              topologyKey: kubernetes.io/hostname
              labelSelector:
                matchLabels:
                  app.kubernetes.io/name: chatcli
                  app.kubernetes.io/instance: chatcli-prod
    # nodeSelector: {}
    # tolerations: []
    # priorityClassName: <an existing PriorityClass>

  # For provider BEDROCK on EKS (IRSA): the operator merges these annotations
  # into the Instance's ServiceAccount.
  # serviceAccount:
  #   annotations:
  #     eks.amazonaws.com/role-arn: arn:aws:iam::<account-id>:role/<role-name>

  # extraEnv: []   # extra variables for the server container; the log
  #                # already reaches `kubectl logs` (JSON lines on stderr)

kubectl apply -f instance.yaml
kubectl -n chatcli get instance chatcli-prod -w

What the operator creates, and what it does not:

A Deployment chatcli-prod whose pods run as non-root with a read-only root filesystem, drop all capabilities, and meet the restricted Pod Security Standard. Startup, readiness and liveness probes call GET /healthz on the metrics port.
A Service chatcli-prod with ports grpc and metrics. With replicas > 1 it is headless, and the operator balances over the pods.
A ServiceAccount, and for watcher targets in other namespaces a ClusterRoleBinding to chatcli-watcher, so the server can read those workloads.
No PodDisruptionBudget, HPA or NetworkPolicy. Create them yourself if you need them (a NetworkPolicy example follows).

Persistence and replicas. persistence.enabled: true creates one ReadWriteOnce PVC (chatcli-prod-sessions) that every replica mounts. The access mode cannot be changed. With replicas > 1, pods scheduled on different nodes stay ContainerCreating with a Multi-Attach error. Either keep replicas: 1 with persistence, or run several replicas without it, as above. With persistence the operator sets the Deployment strategy to Recreate, so the old pod releases the volume before the new one starts: each rollout has a brief downtime. Without it the strategy is RollingUpdate. The AIOps pipeline does not need the session volume. Without it, sessions of remote chatcli connect users live only as long as the pod does.

Fallback. The server builds its fallback chain only when at least two of the listed providers have credentials, and uses it only for requests that bring no credentials or provider of their own. The AIOps pipeline’s requests qualify.

A NetworkPolicy for the Instance, allowing gRPC from the operator’s namespace and scrapes from monitoring (add the namespaces of your chatcli connect users):

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: chatcli-prod
  namespace: chatcli
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/name: chatcli
      app.kubernetes.io/instance: chatcli-prod
  policyTypes: [Ingress]
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: chatcli-system
      ports:
        - port: 50051
          protocol: TCP
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: monitoring
      ports:
        - port: 9090          # metrics and /healthz, plain HTTP without authentication
          protocol: TCP

Kubelet probes are exempt from NetworkPolicy on most CNIs, but not all. If the pods fail readiness after you apply the policy, also allow port 9090 from your node CIDR.

Check: every condition is True and ServerReachable reads Serving:

kubectl -n chatcli get instance chatcli-prod \
  -o jsonpath='{range .status.conditions[*]}{.type}={.status} {.reason}: {.message}{"\n"}{end}'

AuthenticationConfigured=True CredentialConfigured: ...
OperatorCredentialConfigured=True CredentialAvailable: operator authenticates with spec.server.token
TLSConfigured=True SecretConfigured: server certificate from Secret chatcli-prod-tls
Available=True DeploymentReady: All replicas are ready
ServerReachable=True Serving: server <version> serving (CLAUDEAI/claude-sonnet-5)

The VERSION column of kubectl get instance shows the server version the operator last read. A ready Instance is re-probed every five minutes, so allow up to five minutes for ServerReachable to reflect a change; after a failed probe it is retried every 30 seconds.

4. Link source repositories (optional)

A SourceRepository gives the AI recent commits and code around stack traces for one workload. Create it, and its git Secret, in the workload’s namespace. The operator clones it into its writable /tmp emptyDir (chart value tmpVolume.sizeLimit, step 1).

kubectl -n production create secret generic api-gateway-git \
  --from-literal=token='<read-only-git-token>'

apiVersion: platform.chatcli.io/v1alpha1
kind: SourceRepository
metadata:
  name: api-gateway
  namespace: production
spec:
  url: "https://git.example.com/platform/api-gateway.git"
  branch: main
  authType: token            # the Secret key must be named "token"
  secretRef: api-gateway-git
  resource:
    kind: Deployment
    name: api-gateway
    namespace: production
  paths: ["cmd/", "internal/"]
  language: Go
  syncIntervalMinutes: 30

The Secret keys depend on authType: token for token, username and password for basic, ssh-key and known_hosts for ssh. HTTPS credentials reach git through a GIT_ASKPASS helper for each command and are never written to the clone’s .git/config; a clone made by an earlier version, which kept the token in the origin URL, has it stripped from origin and its reflog expired on the next sync. The operator image ships openssh-client, and an SSH clone checks the server’s host key against the Secret’s known_hosts with StrictHostKeyChecking=yes:

ssh-keyscan git.example.com > known_hosts
kubectl -n production create secret generic api-gateway-git \
  --from-file=ssh-key=./deploy_key --from-file=known_hosts=./known_hosts

Without known_hosts an SSH sync fails (secret ... has no "known_hosts" key) unless the SourceRepository sets spec.sshHostKeyPolicy: acceptNew, which trusts the key the host presents on first contact and rejects a different key afterwards (remembered for the life of the operator pod). The default, strict, is the safe choice: pin the host keys in the Secret.

Check: kubectl -n production get sourcerepositories shows the repository ready after the first sync.

5. Notifications

Put the webhook URLs, routing keys and passwords in a Secret. Every key of a channel’s secretRef Secret is merged into its config. Keep them out of config itself: GET /api/v1/policies/notification returns the whole spec to any key with the viewer role.

kubectl -n chatcli-system create secret generic notify-slack \
  --from-literal=webhook_url='https://hooks.slack.com/services/<id>'
kubectl -n chatcli-system create secret generic notify-pagerduty \
  --from-literal=routing_key='<pagerduty-routing-key>'
kubectl -n chatcli-system create secret generic notify-smtp \
  --from-literal=smtp_user='aiops@example.com' \
  --from-literal=smtp_password='<smtp-password>'

apiVersion: platform.chatcli.io/v1alpha1
kind: NotificationPolicy
metadata:
  name: production-notifications
  namespace: chatcli-system
spec:
  enabled: true
  channels:
    - name: slack-incidents
      type: slack
      config:
        channel: "#incidents"
      secretRef:
        name: notify-slack
    - name: pagerduty-critical
      type: pagerduty
      config: {}
      secretRef:
        name: notify-pagerduty
    - name: email-sre
      type: email
      config:
        smtp_host: smtp.example.com
        smtp_port: "587"                 # config is a map of strings: quote numbers
        from: aiops@example.com
        to: "sre@example.com,oncall@example.com"
      secretRef:
        name: notify-smtp
  rules:
    - name: critical-page
      severities: [critical]
      namespaces: [production]           # the affected resource's namespace
      states: [Detected, Contained, Escalated, Resolved]
      channels: [pagerduty-critical, slack-incidents]
    - name: high-to-slack
      severities: [critical, high]
      states: [Detected, Remediating, Contained, Resolved, Escalated, Failed]
      channels: [slack-incidents]
    - name: slo-violations
      signalTypes: [slo_violation]       # Issues opened by the SLO controller
      channels: [slack-incidents, email-sre]
  throttle:
    maxPerHour: 20
    deduplicationWindow: "30s"

Keep deduplicationWindow short. A notification inside the window is dropped, not queued, so with the default 5m an Issue that resolves quickly may never send Resolved to Slack or email. Resolved to PagerDuty and OpsGenie is never throttled, by the window or by maxPerHour, so the page always closes. Contained means the platform silenced the impact (for example, scaled a workload to zero) without fixing the cause, so a human must act: route it to your pager. See Notifications for every channel’s keys and the templates.

6. Escalation

An escalation starts when an Issue reaches Escalated, which happens when remediation attempts run out. It stops when the Issue is Resolved or when someone acknowledges it. The operator uses the first enabled policy, from any namespace, whose severities include the Issue’s.

apiVersion: platform.chatcli.io/v1alpha1
kind: EscalationPolicy
metadata:
  name: production-escalation
  namespace: chatcli-system
spec:
  enabled: true
  severities: [critical, high]
  levels:
    - name: L1-OnCall
      timeoutMinutes: 10
      targets:
        - type: oncall
          name: sre-primary
      notifyChannels: [slack-incidents]
    - name: L2-SRE-Leads
      timeoutMinutes: 20
      targets:
        - type: team
          name: sre-leads
        - type: user
          name: sre-lead@example.com
      notifyChannels: [slack-incidents, pagerduty-critical, email-sre]
      repeatIntervalMinutes: 15          # re-page every 15 minutes until acknowledged or resolved

targets are informational: they are written into the message, and the message goes to each level’s notifyChannels, looked up by name in the enabled NotificationPolicies.

How an escalation behaves:

The current level is saved on the Issue (platform.chatcli.io/escalation-level, with escalation-notified-at for the last page). Each timeoutMinutes moves it one level up, until the last level, where it stays. chatcli_operator_escalation_level_reached goes up by one for every level reached.
repeatIntervalMinutes re-sends the current level’s message at that interval; on the last level it keeps repeating. 0 (the default) never repeats.
Acknowledge (dashboard, or POST /api/v1/incidents/<name>/acknowledge) stops the escalation where it is: no further level, no repeat. The EscalationPolicy records acknowledgedAt and acknowledgedBy (the caller’s API key role) in status.activeEscalations.
Snooze (POST /api/v1/incidents/<name>/snooze with a positive duration, such as "30m") holds every notification of the Issue except Resolved, and holds the escalation. A level page that falls inside the snooze is marked escalation-pending-notify and goes out when the snooze ends; the level’s timer restarts at that moment.
Issues from a chaos drill never escalate.

7. SLOs and SLAs

SLO

SLOs are computed from the operator’s own Issues and Anomalies for the resource, not from Prometheus. Create the SLO in the workload’s namespace.

apiVersion: platform.chatcli.io/v1alpha1
kind: ServiceLevelObjective
metadata:
  name: api-gateway-availability
  namespace: production
spec:
  serviceName: api-gateway
  description: "api-gateway keeps 99.9% availability over 30 days"
  enabled: true
  indicator:
    type: availability
    metricSource: issues
    resource:
      kind: Deployment
      name: api-gateway
      namespace: production
  target:
    percentage: 99.9
    window: "30d"
  alertPolicy:
    burnRateWindows:
      - shortWindow: "1h"
        longWindow: "6h"
        burnRateThreshold: 14.4
        severity: critical
      - shortWindow: "6h"
        longWindow: "72h"
        burnRateThreshold: 6
        severity: high
    pageOnBudgetExhausted: false

A firing burn-rate window opens an Issue with signalType: slo_violation, which the slo-violations rule from step 5 routes.

pageOnBudgetExhausted: true opens one critical Issue when the budget reaches zero. It pages again only after the budget has climbed back above zero and run out again; while that exhaustion Issue is still open, no second one is created.
The SLO’s own slo_violation Issues do not count as downtime, so a page does not deepen the breach it reports.
metricSource: prometheus is not queried: the SLI is still computed from Issues and Anomalies, and the SLO reports the condition MetricSourceSupported=False. prometheusQuery, latencyPercentile and latencyThresholdMs are not evaluated, and alertPolicy.notificationPolicyRef is reserved: SLO Issues are notified by every NotificationPolicy whose rules match them.

SLA

An IncidentSLA in the Issue’s namespace wins. Otherwise, one with the same severity from any namespace applies, so a set in chatcli-system works as the cluster-wide default.

apiVersion: platform.chatcli.io/v1alpha1
kind: IncidentSLA
metadata:
  name: sla-critical
  namespace: chatcli-system
spec:
  severity: critical
  responseTime: "5m"
  resolutionTime: "1h"
  businessHoursOnly: false        # critical runs on a 24/7 clock
---
apiVersion: platform.chatcli.io/v1alpha1
kind: IncidentSLA
metadata:
  name: sla-high
  namespace: chatcli-system
spec:
  severity: high
  responseTime: "15m"
  resolutionTime: "4h"            # whole days may lead the value: "1d", "2d12h"
  businessHoursOnly: true
  businessHours:
    timezone: "America/Sao_Paulo"
    startHour: 9
    endHour: 18
    workDays: ["Monday", "Tuesday", "Wednesday", "Thursday", "Friday"]

A breach is recorded in the IncidentSLA status, on the Issue (platform.chatcli.io/sla-violated, plus platform.chatcli.io/sla-name naming the SLA that counted it) and in the audit trail. status.activeViolations goes back down when the Issue resolves. An invalid duration sets the SLA’s Ready condition to False with reason InvalidDuration. A breach does not send a notification: escalationPolicyRef and notificationPolicyRef are reserved and not read. Alert on chatcli_operator_sla_violations_total instead. See SLOs and SLAs for the calculation and the status fields.

8. Dashboard and REST API keys

The REST API and the web dashboard share port 8090 on the operator Service. Every /api/ call needs an X-API-Key header. Keys come from the Secret chatcli-operator-secrets (key api-keys) in the operator’s namespace. The name is fixed. Without it, every /api/ call returns 401.

ADMIN_KEY="$(openssl rand -hex 32)"
VIEWER_KEY="$(openssl rand -hex 32)"

kubectl -n chatcli-system create secret generic chatcli-operator-secrets \
  --from-literal=api-keys="$(cat <<EOF
- key: "${ADMIN_KEY}"
  role: admin
  name: alice
  description: platform team
- key: "${VIEWER_KEY}"
  role: viewer
  description: read-only dashboards
EOF
)"

Role	Can
`viewer`	Read everything (incidents, SLOs, approvals, postmortems, analytics, audit, policies)
`operator`	Also acknowledge, snooze and resolve incidents, approve or reject requests, review postmortems, create and edit runbooks
`admin`	Everything, including deleting runbooks

Any other role string gets no access at all. The operator re-reads the Secret every 30 seconds, so key changes need no restart. Editing the Secret to remove a key revokes it, and an empty list leaves no valid key. YAML that does not parse keeps the last valid key set in force (the operator logs it once per version), so fix the typo rather than relying on it to revoke. A Secret without an api-keys entry falls back to the ConfigMap; deleting both the Secret and the ConfigMap chatcli-operator-config revokes every key within about 30 seconds (401 from then on). A read error other than not found keeps the keys in force. Startup applies the same rules as the poll. Each entry also accepts an optional name, the identity recorded on the approvals that key makes (falling back to description, then to a key-<hash> fingerprint). A quorum counts each key once, so give every approver their own key. Instead of kubectl, the chart can render the same Secret (--set apiKeys.create=true with apiKeys.entries), but the keys are then stored in the Helm release. Open the dashboard:

kubectl -n chatcli-system port-forward svc/chatcli-operator 8090:8090
# http://localhost:8090 — paste a key when asked; the browser keeps it in localStorage

To publish it, serve it at the root path of its own host. The dashboard calls /api/v1/... with an absolute path, so a sub-path with rewrite-target loads the page but breaks every API call.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: chatcli-dashboard
  namespace: chatcli-system
spec:
  ingressClassName: nginx
  tls:
    - hosts: [aiops.example.com]
      secretName: aiops-example-com-tls
  rules:
    - host: aiops.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: chatcli-operator
                port:
                  name: api

The REST rate limit counts requests without a valid key per client host (30 per minute), and each valid key on its own (600 per minute). Behind an ingress, the client host is the ingress controller, so all unauthenticated traffic shares one bucket. For TLS on port 8090 itself, mount a certificate with extraVolumes/extraVolumeMounts and set security.apiTLS.certFile/keyFile. There is no SSO or OIDC login: access is by API key only.

Notes for other ingress controllers, TLS served by the operator itself and local clusters are in Web Dashboard: Accessing and publishing.

9. Approvals

An ApprovalPolicy gates RemediationPlans in its own namespace. Within a policy, the first matching rule wins. A rule’s actionTypes match when any action in the plan is listed. Put the strict rules first, and end with a catch-all.

apiVersion: platform.chatcli.io/v1alpha1
kind: ApprovalPolicy
metadata:
  name: production-approvals
  namespace: production
spec:
  enabled: true
  rules:
    - name: node-actions-manual
      match:
        actionTypes: [DrainNode, CordonNode, UncordonNode]
      mode: manual
      timeoutMinutes: 30
    - name: risky-changes-quorum
      match:
        severities: [critical, high]
        actionTypes: [RollbackDeployment, ScaleDeployment, HelmRollback, ApplyManifest, AdjustResources, PatchConfig]
      mode: quorum
      requiredApprovers: 2
      timeoutMinutes: 20
      changeWindow:
        timezone: "America/Sao_Paulo"
        allowedDays: ["Monday", "Tuesday", "Wednesday", "Thursday", "Friday"]
        startHour: 9
        endHour: 18                      # exclusive
    - name: risky-changes-manual
      match:
        actionTypes: [RollbackDeployment, ScaleDeployment, HelmRollback, ApplyManifest, AdjustResources, PatchConfig]
      mode: manual
      timeoutMinutes: 30
    - name: low-risk-auto
      match:
        severities: [low, medium]
        actionTypes: [RestartDeployment, DeletePod]
      mode: auto
    - name: everything-else
      match: {}
      mode: manual
      timeoutMinutes: 30

How to read it:

auto without autoApproveConditions means the policy does not gate the plan. An auto rule with autoApproveConditions parks the plan and approves it automatically only when every condition holds (maxSeverity is compared with the Issue severity; an unknown severity does not meet it); otherwise a human decides. A plan that contains a restart and an action no earlier rule lists also runs under low-risk-auto.
defaultMode is informational: kubectl get ap shows it, but the operator does not apply it. Without a matching rule the plan is not gated by the policy, which is why the last rule (match: {}) is a catch-all that gates everything else.
DrainNode cordons the node and evicts its pods through the policy/v1 Eviction API with a 30-second grace period, so PodDisruptionBudgets are honored. Evictions refused with 429 or a server error are retried every 5 seconds until the optional timeout param (default 2m, at most 10m); after that the action fails, naming the pods it could not evict. DaemonSet and mirror pods are skipped. It still belongs behind a manual rule.
The gate fails closed. If the Issue, the AIInsight or the ApprovalPolicies cannot be read, or the decision engine errors, the plan stays Pending, a Warning Event ApprovalGateUnavailable is recorded on it, and it is retried with backoff. A plan whose parent Issue is gone fails (“Parent issue not found…”). A missing AIInsight counts as confidence 0, which only makes the gate stricter.

Approve with an annotation on the ApprovalRequest (named approval-<plan>, in the plan’s namespace). Restrict update/patch on approvalrequests with RBAC to the people allowed to approve, because the approver name is free text:

kubectl -n production get approvalrequests
kubectl -n production annotate approvalrequest approval-<plan> \
  platform.chatcli.io/approve="alice:rollback reviewed"

The dashboard and POST /api/v1/approvals/{name}/approve (or /reject) record one decision on a Pending request, the same way the annotation does. The approver is stored as <typed name> (api-key: <key identity>), and the ApprovalReconciler then evaluates requiredApprovers and the change window. Over REST a quorum counts distinct API keys, so a shared key cannot satisfy requiredApprovers: 2; annotation approvers are counted by name, case-insensitively. Any rejection rejects the request. A request that is not Pending, or a second decision by the same key, answers 409.

The change window only gates when a decision takes effect: decisions are recorded at any time, and a rejection applies immediately. The timeout clock runs only while the window is open, and a request already approved that waits only for the window does not expire; it turns Approved when the window opens, with the condition ChangeWindow=False (reason OutsideChangeWindow) meanwhile. A window that can never open blocks approval and expires on wall-clock time. Two more gates can park a plan without any policy: the cluster tier (chart clusterName, naming a ClusterRegistration) and the decision engine (chart decisionEngine.enabled). See Approval Workflow and Decision Engine.

10. Prometheus

Two separate integrations: Scraping. The operator exposes /metrics on port 8080 (chatcli_operator_*). Each Instance exposes /metrics on its metrics port (chatcli_grpc_*, chatcli_llm_*, chatcli_watcher_*, chatcli_server_*). Both are plain HTTP without authentication, so restrict who can reach them with NetworkPolicy (steps 1 and 3). The chart’s serviceMonitor.enabled covers the operator. For the Instance:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: chatcli-prod
  namespace: chatcli
  labels:
    release: kube-prometheus-stack
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: chatcli
      app.kubernetes.io/instance: chatcli-prod
  endpoints:
    - port: metrics
      path: /metrics
      interval: 30s

deploy/grafana/dashboards-configmap.yaml holds only two ServiceMonitors, not the dashboards. Its server ServiceMonitor selects app.kubernetes.io/name: chatcli and the port metrics, which both Instance Services (operator) and the server Helm chart carry, in any namespace; apply it, or keep the narrower Instance ServiceMonitor above.

Queries during analysis. With prometheusUrl set on the operator chart (environment variable PROMETHEUS_URL), the operator queries Prometheus for the incident’s time range and adds the results to the AI analysis: CPU, memory, restarts and network per pod, request and error rates, latency percentiles and HPA replicas. It calls /api/v1/query_range with no authentication, so the URL must reach an unauthenticated Prometheus HTTP API, usually the in-cluster Service. Container metrics come from cAdvisor and restarts from kube-state-metrics. Without it, analysis still runs, without these metrics.

11. Grafana dashboards

The repository ships four dashboards in deploy/grafana/. Their panels reference the datasource ${DS_PROMETHEUS}, which the Grafana import dialog resolves, but sidecar provisioning does not. For the sidecar, substitute your Prometheus datasource UID first (prometheus in kube-prometheus-stack; check Connections → Data sources in Grafana):

git clone --depth 1 --branch v1.226.0 https://github.com/diillson/chatcli.git
mkdir -p chatcli-dashboards
for f in chatcli/deploy/grafana/*.json; do
  sed 's/\${DS_PROMETHEUS}/prometheus/g' "$f" > "chatcli-dashboards/$(basename "$f")"
done

kubectl -n monitoring create configmap chatcli-grafana-dashboards \
  --from-file=chatcli-dashboards/ --dry-run=client -o yaml | kubectl apply -f -
kubectl -n monitoring label configmap chatcli-grafana-dashboards grafana_dashboard=1 --overwrite

Or import each JSON file with Dashboards → New → Import and pick the datasource there. Every panel queries a metric the operator or the server exports; see Web Dashboard for what each dashboard shows.

12. Validate with a chaos drill

A drill proves the loop end to end: a failure is injected, the watcher sees it, an Issue is opened and labeled as a drill, and the workload recovers.

Run drills only against workloads with enough replicas. Nothing is protected by default: every guardrail below is opt-in, and no namespace is blocked unless you list it.

apiVersion: platform.chatcli.io/v1alpha1
kind: ChaosExperiment
metadata:
  name: api-gateway-pod-kill
  namespace: production               # the target's namespace
spec:
  experimentType: pod_kill
  target:
    kind: Deployment
    name: api-gateway
    namespace: production
  duration: "3m"                      # observation window after the single injection
  parameters:
    count: "1"
  dryRun: true                        # run the safety checks only, first
  enabled: true
  safetyChecks:
    minHealthyPods: 2                 # checked once, before injection
    maxConcurrentExperiments: 1
    abortOnIssueDetected: false       # true would abort as soon as AIOps detects the failure you injected
    blockedNamespaces: ["kube-system", "chatcli-system", "chatcli"]
  postExperiment:
    verifyRecovery: true
    recoveryTimeout: "5m"

kubectl apply -f chaos.yaml
kubectl -n production get chaosexperiment api-gateway-pod-kill -w
kubectl -n production get chaosexperiment api-gateway-pod-kill -o jsonpath='{.status.result}{"\n"}'
# dry-run: would execute pod_kill on production/api-gateway for 3m

# Completed is final: editing the object does not start a new run.
# Delete it, set dryRun: false in chaos.yaml, and create it again.
kubectl -n production delete chaosexperiment api-gateway-pod-kill
kubectl apply -f chaos.yaml

What to look for:

# Issues raised during the drill carry the chaos label and never escalate
kubectl -n production get issues -l platform.chatcli.io/source=chaos-experiment
kubectl -n production get chaosexperiment api-gateway-pod-kill \
  -o jsonpath='{.status.state} recovered={.status.recoveryVerified}{"\n"}'

Limitations to know before you plan drills:

pod_kill and pod_failure work with the RBAC the chart ships. cpu_stress, memory_stress and disk_stress create a stress pod, which the chart’s RBAC allows; the pod passes the restricted Pod Security Standard (UID/GID 65534, non-root, RuntimeDefault seccomp, no privilege escalation, all capabilities dropped, read-only root filesystem with an emptyDir at /tmp, no ServiceAccount token). network_delay and network_loss are not supported: the experiment fails in Pending with a message that says nothing was injected.
requireApproval: true raises an ApprovalRequest chaos-<experiment> (requested action Custom, 30-minute timeout). The experiment runs once it is approved; a rejection or expiry fails the experiment.
schedule is not supported: an experiment that sets it fails in Pending and nothing runs. Create one ChaosExperiment per run.
recoveryTime is measured from the end of the injection to the first passing health check (checks run every 5 seconds); recoveryVerified is the pass/fail signal. Issues from the drill are left out of MTTD and MTTR.

See Chaos Engineering for every type, parameter and state.

13. Verify the installation

Operator and CRDs

kubectl -n chatcli-system get pods -l app.kubernetes.io/name=chatcli-operator
kubectl get crd -o name | grep -c platform.chatcli.io            # 17

Instance conditions

kubectl -n chatcli get instance chatcli-prod
kubectl -n chatcli get instance chatcli-prod \
  -o jsonpath='{range .status.conditions[*]}{.type}={.status} {.reason}{"\n"}{end}'

All five conditions must be True, as in step 3.

The alert bridge is connected

kubectl -n chatcli-system logs -l app.kubernetes.io/name=chatcli-operator --prefix --tail=-1 \
  | grep -E 'Connected to Instance|Alert stream open|no ready Instance'

Expect Connected to Instance followed by Alert stream open on the leader. Connected to Instance alone proves nothing, because the connection is opened lazily. TLS and credential errors surface in ServerReachable and in the stream errors after it.

gRPC end to end, with TLS and the token

kubectl -n chatcli port-forward svc/chatcli-prod 50051:50051 &
kubectl -n chatcli get secret chatcli-prod-tls -o jsonpath='{.data.ca\.crt}' | base64 -d > ca.crt
TOKEN="$(kubectl -n chatcli get secret chatcli-prod-token -o jsonpath='{.data.token}' | base64 -d)"

chatcli connect --addr localhost:50051 --tls --ca-cert ca.crt --token "$TOKEN" -p "Reply with OK"

This relies on the localhost SAN from step 2.

REST API and dashboard

kubectl -n chatcli-system port-forward svc/chatcli-operator 8090:8090 &
curl -s http://localhost:8090/readyz                                   # {"status":"ready",...}
curl -s -o /dev/null -w '%{http_code}\n' http://localhost:8090/api/v1/incidents                        # 401
curl -s -o /dev/null -w '%{http_code}\n' -H "X-API-Key: $VIEWER_KEY" http://localhost:8090/api/v1/incidents  # 200

Metrics

In Prometheus, up{namespace="chatcli-system"} and up{namespace="chatcli"} are 1, and chatcli_operator_instance_ready and chatcli_server_info return series.

Policies are loaded

kubectl get notificationpolicies,escalationpolicies,incidentslas -A
kubectl -n production get approvalpolicies,servicelevelobjectives

The pipeline reacts

Run the drill from step 12 and watch an Issue appear in production and in the dashboard.

14. Troubleshooting

Symptom	Cause	Fix
No Deployment; `AuthenticationConfigured=False` (`CredentialMissing`)	The Instance has no credential	Set `spec.server.token` (or JWT, or a client CA)
No Deployment; `TLSConfigured=False` (`SecretNameMissing`)	`tls.enabled: true` without `tls.secretName`	Set the Secret name
Pods stuck in `ContainerCreating`, event `secret "..." not found`	The TLS Secret does not exist in the Instance’s namespace	Create it (step 2)
Pod exits at startup	The refusal is the last line of `kubectl logs --previous` (the server writes its log to stderr in a pod); most often a missing credential	Read it and fix what it names
Log shows `refusing to serve an unauthenticated API`	The token Secret or key is missing. Secret references are optional, so the variable is silently dropped	Check the Secret name and the `key` in `spec.server.token`
`ServerReachable=False`, `x509: certificate is valid for ..., not chatcli-prod.chatcli.svc.cluster.local`	The certificate lacks the in-cluster name	Reissue it with the SANs from step 2
`ServerReachable=False`, `x509: certificate signed by unknown authority`	No `ca.crt` in the Secret, and the CA is not public	Add `ca.crt` (cert-manager with a CA issuer writes it)
`ServerReachable=False`, `server info (credential check)` and `Unauthenticated`	The operator’s credential is rejected	Check `OperatorCredentialConfigured`; an RS256-only (`jwtPublicKeyRef`) server needs `operatorTokenRef` or `operatorClientCertSecretName`
`Unauthenticated: authentication failed` and the server log shows `auth failure rate limit exceeded`	Something on the operator’s host failed authentication more than 5 times in a minute and exhausted that host’s failure budget (burst 5, then one every 12 seconds; only failed authentications count, valid credentials are never throttled)	Find the caller with the wrong or expired credential; the budget refills on its own
`ServerReachable=False` (`ProbeFailed`, `DeadlineExceeded` or `connection refused`) right after a rollout, while the pods are Ready	The probe ran while the pods were being replaced	Transient: a failed probe is retried after 30 seconds; see Instance conditions
`ServerReachable=Unknown` (`DeploymentNotReady`)	No pod is ready	`kubectl -n chatcli describe pod`; check the probes on `/healthz` (port 9090)
Some pods `ContainerCreating` with `Multi-Attach error`	RWO sessions PVC with `replicas > 1`	Disable persistence or use one replica (step 3)
Operator logs `no ready Instance found`	No Instance is `Ready` yet	Fix the Instance first
Dashboard or `/api/` returns 401 `no API keys configured`	The Secret `chatcli-operator-secrets` is missing, or in another namespace	Create it in the operator’s namespace (step 8)
REST returns 429	Rate limit: 30 per minute per host without a valid key	Send a valid `X-API-Key`; each key gets 600 per minute
Dashboard page loads behind an ingress, but every panel errors	Served under a sub-path	Use a dedicated host at `/` (step 8)
No Issues for a watched workload	The target is in another namespace without RBAC, or several Instances are ready	Check the `chatcli-chatcli-prod-watcher` ClusterRoleBinding; keep one Instance per cluster
`SourceRepository` sync fails with `read-only file system` or `no space left on device`	An operator chart without the `/tmp` emptyDir, or a clone larger than `tmpVolume.sizeLimit`	Upgrade the operator chart; raise `tmpVolume.sizeLimit`
`SourceRepository` SSH sync fails with `has no "known_hosts" key` or `Host key verification failed`	The Secret has no `known_hosts`, or the host key changed	Add `ssh-keyscan` output as `known_hosts`, or set `spec.sshHostKeyPolicy: acceptNew`
Grafana: `Datasource ${DS_PROMETHEUS} was not found`	Provisioned through the sidecar without substituting the datasource UID	Use the `sed` loop from step 11, or import through the UI
Chaos experiment stays `Pending`	`requireApproval: true`, or `maxConcurrentExperiments` reached	Remove `requireApproval`; wait for the running experiment

Get started

Documentation

AIOps: Complete Production Setup

Prerequisites

Where each resource lives

1. Install the operator

2. Create the Instance Secrets

TLS certificate

3. Create the production Instance

4. Link source repositories (optional)

5. Notifications

6. Escalation

7. SLOs and SLAs

SLO

SLA

8. Dashboard and REST API keys

9. Approvals

10. Prometheus

11. Grafana dashboards

12. Validate with a chaos drill

13. Verify the installation

14. Troubleshooting

Production checklist

Next steps

Security hardening

K8s Operator

Approval Workflow

REST API

​Prerequisites

​Where each resource lives

​1. Install the operator

​2. Create the Instance Secrets

​TLS certificate

​3. Create the production Instance

​4. Link source repositories (optional)

​5. Notifications

​6. Escalation

​7. SLOs and SLAs

​SLO

​SLA

​8. Dashboard and REST API keys

​9. Approvals

​10. Prometheus

​11. Grafana dashboards

​12. Validate with a chaos drill

​13. Verify the installation

​14. Troubleshooting

​Production checklist

​Next steps

Security hardening

K8s Operator

Approval Workflow

REST API

Prerequisites

Where each resource lives

1. Install the operator

2. Create the Instance Secrets

TLS certificate

3. Create the production Instance

4. Link source repositories (optional)

5. Notifications

6. Escalation

7. SLOs and SLAs

SLO

SLA

8. Dashboard and REST API keys

9. Approvals

10. Prometheus

11. Grafana dashboards

12. Validate with a chaos drill

13. Verify the installation

14. Troubleshooting

Production checklist

Next steps