Kubernetes Operator (AIOps)

​

How it fits together

• An Instance is one ChatCLI gRPC server (chatcli server) that the operator deploys and keeps in shape: a Deployment, a Service, ConfigMaps, a ServiceAccount and optionally a PVC and watcher RBAC.
• The operator always dials the server over TLS 1.3. There is no plaintext path from the operator to an Instance. The address it dials is <name>.<namespace>.svc.cluster.local:<port>, so the server certificate must be valid for that name.
• Every Instance needs a credential. Inside a cluster the server listens on every interface and refuses to start without a shared token, JWT material or a client CA. The operator checks this up front and does not create the Deployment when the spec has none.
• One Instance per cluster drives AIOps. The WatcherBridge lists every Instance in the cluster and attaches to the first ready one it finds. You can run more Instances for chat traffic, but only one feeds the incident pipeline, and which one is not something to rely on.
• The REST API and web dashboard are served by the operator, not by the Instance: Service chatcli-operator, port 8090, authenticated with API keys.

​

API group and CRDs

​

Prerequisites

​

Install the operator

helm install chatcli-operator oci://ghcr.io/diillson/charts/chatcli-operator \
  --version 1.214.0 \
  -n chatcli-system --create-namespace

cd operator
make deploy

​

Verify the install

kubectl get crd -o name | grep -c 'platform.chatcli.io'   # 17
kubectl -n chatcli-system get deploy,svc,pods
kubectl -n chatcli-system logs deploy/chatcli-operator | grep -E 'starting manager|REST|allowlist'

​

Your first Instance

kubectl create namespace chatcli

kubectl -n chatcli create secret generic chatcli-api-keys \
  --from-literal=ANTHROPIC_API_KEY='<your-anthropic-key>'

kubectl -n chatcli create secret generic chatcli-server-token \
  --from-literal=token="$(openssl rand -hex 32)"

NAME=chatcli NS=chatcli

cat > ca.cnf <<EOF
[req]
distinguished_name = dn
prompt = no
x509_extensions = v3_ca
[dn]
CN = chatcli-internal-ca
[v3_ca]
basicConstraints = critical, CA:TRUE
keyUsage = critical, keyCertSign, cRLSign
subjectKeyIdentifier = hash
EOF

cat > server.cnf <<EOF
[req]
distinguished_name = dn
prompt = no
req_extensions = v3_req
[dn]
CN = ${NAME}.${NS}.svc.cluster.local
[v3_req]
basicConstraints = critical, CA:FALSE
keyUsage = critical, digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth
subjectAltName = @alt_names
[alt_names]
DNS.1 = ${NAME}.${NS}.svc.cluster.local
DNS.2 = ${NAME}.${NS}.svc
DNS.3 = ${NAME}.${NS}
DNS.4 = ${NAME}
DNS.5 = localhost
IP.1  = 127.0.0.1
EOF

# 1. The CA (keep ca.key safe: it signs every certificate you trust)
openssl req -x509 -new -nodes -newkey rsa:4096 -sha256 -days 1825 \
  -keyout ca.key -out ca.crt -config ca.cnf

# 2. The server key, CSR and certificate signed by the CA
openssl req -new -nodes -newkey rsa:2048 \
  -keyout tls.key -out tls.csr -config server.cnf
openssl x509 -req -in tls.csr -CA ca.crt -CAkey ca.key -CAcreateserial \
  -out tls.crt -days 825 -sha256 -extfile server.cnf -extensions v3_req

# 3. Check the SANs
openssl x509 -in tls.crt -noout -ext subjectAltName

# 4. The Secret: tls.crt and tls.key for the server, ca.crt for the operator
kubectl -n chatcli create secret generic chatcli-tls \
  --from-file=tls.crt --from-file=tls.key --from-file=ca.crt

apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: chatcli-selfsigned
  namespace: chatcli
spec:
  selfSigned: {}
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: chatcli-ca
  namespace: chatcli
spec:
  isCA: true
  commonName: chatcli-internal-ca
  secretName: chatcli-ca
  duration: 43800h   # 5 years
  privateKey:
    algorithm: ECDSA
    size: 256
  issuerRef:
    name: chatcli-selfsigned
    kind: Issuer
---
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: chatcli-ca
  namespace: chatcli
spec:
  ca:
    secretName: chatcli-ca
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: chatcli-tls
  namespace: chatcli
spec:
  secretName: chatcli-tls
  duration: 2160h     # 90 days
  renewBefore: 360h   # 15 days
  usages: ["digital signature", "key encipherment", "server auth"]
  dnsNames:
    - chatcli.chatcli.svc.cluster.local
    - chatcli.chatcli.svc
    - chatcli.chatcli
    - chatcli
    - localhost
  ipAddresses:
    - 127.0.0.1
  issuerRef:
    name: chatcli-ca
    kind: Issuer

apiVersion: platform.chatcli.io/v1alpha1
kind: Instance
metadata:
  name: chatcli
  namespace: chatcli
spec:
  provider: CLAUDEAI
  model: claude-sonnet-5
  apiKeys:
    name: chatcli-api-keys
  server:
    token:
      name: chatcli-server-token
      key: token
    tls:
      enabled: true
      secretName: chatcli-tls
  resources:            # the operator applies none by default
    requests:
      cpu: 250m
      memory: 256Mi
    limits:
      cpu: "1"
      memory: 1Gi

kubectl apply -f instance.yaml

kubectl -n chatcli get instance chatcli

NAME      READY   REPLICAS   PROVIDER   VERSION             AGE
chatcli   true    1          CLAUDEAI   v1.214.0   2m

kubectl -n chatcli get instance chatcli \
  -o jsonpath='{range .status.conditions[*]}{.type}={.status} {.reason}: {.message}{"\n"}{end}'

TLSConfigured=True SecretConfigured: server certificate from Secret chatcli-tls
AuthenticationConfigured=True CredentialConfigured: server has a credential, or binds loopback only
OperatorCredentialConfigured=True CredentialAvailable: operator authenticates with spec.server.token
Available=True DeploymentReady: All replicas are ready
ServerReachable=True Serving: server v1.214.0 serving (CLAUDEAI/claude-sonnet-5)

​

Instance conditions

kubectl -n chatcli annotate instance chatcli chatcli.io/reprobe="$(date +%s)" --overwrite

​

Logs worth reading

# The server: bind address, TLS, auth mode, startup refusals
kubectl -n chatcli logs deploy/chatcli

# The pods: probe failures, image pulls, missing Secret volumes
kubectl -n chatcli describe pod -l app.kubernetes.io/instance=chatcli

# The operator: "instance not provisioned: ...", "Connected to Instance", alert stream state
kubectl -n chatcli-system logs deploy/chatcli-operator

​

Connect the CLI

kubectl -n chatcli port-forward svc/chatcli 50051:50051

TOKEN=$(kubectl -n chatcli get secret chatcli-server-token -o jsonpath='{.data.token}' | base64 -d)
chatcli connect localhost:50051 --tls --ca-cert ca.crt --token "$TOKEN"

# One-shot
chatcli connect localhost:50051 --tls --ca-cert ca.crt --token "$TOKEN" -p "summarize the last deploy"

• Client certificate (mTLS): there is no flag. Set CHATCLI_TLS_CLIENT_CERT and CHATCLI_TLS_CLIENT_KEY; they are used only together with --tls.
• Plaintext: without --tls the CLI still dials TLS with the system CAs unless CHATCLI_ALLOW_INSECURE=true is set. A plaintext server is only useful for local development; the operator cannot reach one.
• The full client reference is in Remote Connect.

​

Exposing the server outside the cluster

• L4 load balancer (for example an AWS NLB): create a Service of type LoadBalancer selecting app.kubernetes.io/name: chatcli and app.kubernetes.io/instance: chatcli, port 50051 to targetPort: grpc, with TCP passthrough. TLS stays end to end, which is also the only way client certificates (mTLS) reach the server.
• ingress-nginx: the backend speaks TLS, so use nginx.ingress.kubernetes.io/backend-protocol: "GRPCS", or TLS passthrough (nginx.ingress.kubernetes.io/ssl-passthrough: "true", which requires the controller flag --enable-ssl-passthrough). A terminating ingress cannot carry mTLS client certificates to the server.
• Clients then connect with chatcli connect chatcli.example.com:443 --tls --token "$TOKEN" (add --ca-cert for a private CA).

​

Dashboard and REST API

KEY=$(openssl rand -hex 32)
kubectl -n chatcli-system create secret generic chatcli-operator-secrets \
  --from-literal=api-keys="$(printf -- '- key: "%s"\n  role: admin\n  description: platform team\n' "$KEY")"

apiVersion: v1
kind: Secret
metadata:
  name: chatcli-operator-secrets   # fixed name
  namespace: chatcli-system        # the operator's namespace
type: Opaque
stringData:
  api-keys: |
    - key: "<openssl rand -hex 32>"
      role: admin
      description: platform team
    - key: "<openssl rand -hex 32>"
      role: viewer
      description: read-only wallboard

apiKeys:
  create: true
  entries:
    - key: "<openssl rand -hex 32>"
      role: admin
      description: platform team

kubectl -n chatcli-system port-forward svc/chatcli-operator 8090:8090
# Dashboard: http://127.0.0.1:8090/  (paste the key; the browser keeps it in local storage)
curl -s -H "X-API-Key: $KEY" http://127.0.0.1:8090/api/v1/incidents

• Publishing it: put an Ingress in front of the Service at the root path of a host of its own. The page calls /api/v1/... with an absolute path, so a sub-path (/chatcli, with a rewrite or strip-path) loads the page and breaks every panel. Examples and controller notes: Web Dashboard: Accessing and publishing.
• Rate limits: 30 requests per minute per client host for requests without a valid key (what bounds key guessing), 600 per minute per valid key. Exceeding them returns 429 with Retry-After.
• TLS: set security.apiTLS.certFile/keyFile and mount the Secret with extraVolumes/extraVolumeMounts; the API then serves HTTPS with TLS 1.3 only.
• Dev mode: security.devMode: true (CHATCLI_OPERATOR_DEV_MODE, read as a boolean: true, TRUE, 1 or t) admits every call as admin when no keys are configured; the startup log reports the same mode the API applies. It exists for local experiments; never enable it on a shared cluster.
• Endpoints, error codes and pagination: REST API reference. Dashboard tour: Web Dashboard.

​

Turn on AIOps

spec:
  watcher:
    enabled: true
    interval: "30s"
    window: "2h"
    maxLogLines: 100
    maxContextChars: 32000
    targets:
      - name: api-gateway            # kind defaults to Deployment
        namespace: production
        metricsPort: 9090
        metricsFilter: ["http_requests_*", "http_request_duration_*"]
      - name: postgres
        kind: StatefulSet
        namespace: production
      - name: fluentd
        kind: DaemonSet
        namespace: logging
      - name: nightly-etl
        kind: CronJob
        namespace: data

1. Detection. The WatcherBridge (on the operator leader) keeps the server’s StreamAlerts stream open (heartbeat every 15s; a server without the RPC, older than 1.211.0, is polled with GetAlerts every 30s, and the stream is retried every 10 minutes or as soon as that server goes away; alertTransport: poll forces polling) and creates Anomaly resources, deduplicated for aiops.dedupTTLMinutes (default 30).
2. Correlation. Anomalies on the same resource are grouped into an Issue with a risk score and severity.
3. Analysis. An AIInsight is created and the server’s AnalyzeIssue RPC asks the LLM for a root cause, enriched with Kubernetes context, logs, metrics, GitOps state and linked source code.
4. Remediation. A RemediationPlan runs a matching Runbook, an AI-generated runbook, or an agentic observe-decide-act loop (AgenticStep RPC), with 54 typed actions (plus Custom, which the safety checks reject), a pre-action snapshot and automatic rollback. ApprovalPolicies, the decision engine and the cluster tier can park a plan for a human.
5. Resolution. The Issue resolves or escalates after aiops.maxRemediationAttempts, and a PostMortem is generated. Notification and escalation are driven by NotificationPolicy and EscalationPolicy.

​

ExecDiagnostic Allowlist

helm upgrade chatcli-operator oci://ghcr.io/diillson/charts/chatcli-operator \
  --version 1.214.0 -n chatcli-system -f operator-values.yaml \
  --set security.allowedDiagnosticCommands="curl -s localhost:5678/health\,nc -zv redis.default.svc.cluster.local 6379"

kubectl -n chatcli-system logs deploy/chatcli-operator | grep 'Effective ExecDiagnostic allowlist loaded'

​

Source repositories

ssh-keyscan github.com > known_hosts
kubectl -n production create secret generic api-server-git \
  --from-file=ssh-key=./deploy_key --from-file=known_hosts=./known_hosts

​

Watching the pipeline

kubectl get anomalies -A            # SOURCE  SIGNAL  CORRELATED  AGE
kubectl get issues -A               # SEVERITY  STATE  RISK  AGE
kubectl get aiinsights -A           # ISSUE  PROVIDER  CONFIDENCE  AGE
kubectl get remediationplans -A     # ISSUE  ATTEMPT  STATE  AGE
kubectl get postmortems -A          # ISSUE  SEVERITY  STATE  AGE
kubectl get approvalrequests -A     # ISSUE  PLAN  STATE  RULE  AGE

​

Authentication options

kubectl -n chatcli create secret generic chatcli-jwt --from-literal=secret="$(openssl rand -hex 48)"

spec:
  server:
    security:
      jwtSecretRef:
        name: chatcli-jwt
        key: secret
      jwtIssuer: "https://auth.example.com"   # optional: required iss
      jwtAudience: "chatcli"                  # optional: required aud

spec:
  server:
    security:
      jwtPublicKeyRef:            # PEM public key(s) the server verifies with
        name: chatcli-jwt-public
        key: public.pem
      operatorTokenRef:           # a JWT your IdP issued for the operator
        name: chatcli-operator-jwt
        key: token

spec:
  server:
    tls:
      enabled: true
      secretName: chatcli-tls
      clientCASecretName: chatcli-client-ca   # Secret with ca.crt
    security:
      mtlsRole: user                          # role of certificate-only callers
      operatorClientCertSecretName: chatcli-operator-client   # tls.crt + tls.key signed by that CA

​

Instance spec reference

​

Top level

​

spec.server

​

spec.server.security

​

spec.watcher

​

spec.fallback

spec:
  provider: CLAUDEAI
  model: claude-sonnet-5
  fallback:
    enabled: true
    providers:            # effective chain: CLAUDEAI, OPENAI, GOOGLEAI
      - name: OPENAI
        model: gpt-6-sol
      - name: GOOGLEAI
        model: gemini-3.8-flash

​

spec.aiops

​

spec.features and spec.pipeline

​

spec.mcp, spec.agents, spec.plugins

​

Status

​

What the operator creates

• strategy Recreate when persistence.enabled (the sessions PVC is ReadWriteOnce, so the old pod releases it before the new one starts; each rollout has a brief downtime), otherwise RollingUpdate;
• probes on GET /healthz of the metrics port (plain HTTP, no credential): startup every 5s up to 5 minutes, readiness every 10s (out of the Service after 30s of failures), liveness every 20s (restart after 2 minutes);
• pod security context runAsNonRoot, UID 1000, fsGroup: 1000 with fsGroupChangePolicy: OnRootMismatch (so a fresh sessions volume is writable whatever the provisioner does), seccompProfile: RuntimeDefault (unless spec.securityContext), and for every container, the plugin-loader init container included, allowPrivilegeEscalation: false, readOnlyRootFilesystem: true, all capabilities dropped: the pod passes the restricted Pod Security Standard;
• writable emptyDirs /tmp (100Mi) and /home/chatcli/.chatcli (200Mi); anything outside the sessions PVC is lost when the pod restarts;
• HOME=/home/chatcli, the <name> ConfigMap and the apiKeys Secret as envFrom, then extraEnv, then the typed credential and feature variables. Secret references are optional, so a missing Secret does not block the pod: it starts without the value.

​

Rollout triggers

​

Running in production

​

Checklist

• The TLS certificate covers <name>.<namespace>.svc.cluster.local (plus the short names, and your external name if exposed), and the Secret has ca.crt for a private CA.
• TLSConfigured, AuthenticationConfigured, OperatorCredentialConfigured, Available and ServerReachable are all True.
• A credential per audience: a shared token only for automation you trust with admin, JWTs (HS256 or RS256) or mTLS for people.
• spec.resources requests and limits are set on every Instance.
• Dashboard API keys live in chatcli-operator-secrets, one per team with the least role it needs, and security.devMode is off.
• NetworkPolicies restrict the Instance and operator pods, including the plain-HTTP metrics ports.
• Images are pinned (operator chart version; spec.image.tag only if you want to decouple the server from the operator release).
• Only one Instance in the cluster is meant to drive AIOps.
• The ExecDiagnostic allowlist covers your workloads’ health ports.
• Custom resources and referenced Secrets are backed up.

​

TLS and RBAC

• TLS everywhere the operator talks. The operator dials Instances with TLS 1.3 only, verifying the certificate against the Instance’s ca.crt (or the system CAs, or security.grpcTLS.caFile). The server itself accepts TLS 1.3 only. Rotate with cert-manager or by replacing the Secret: the pods roll (chatcli.io/tls-hash) and the operator trusts the new ca.crt on its next dial.
• Operator RBAC. The operator’s ClusterRole is broad by necessity: full access to its 17 CRDs; Deployments, Services, ConfigMaps, ServiceAccounts and PVCs; read and write on Secrets cluster-wide (it reads referenced Secrets and executes RotateSecret); pods (including create, for chaos stress pods), pod eviction (DrainNode uses the Eviction API), logs, and core and events.k8s.io events; nodes (cordon/drain); create and update on every kind the ApplyManifest allowlist admits (workloads, Jobs, CronJobs, Services, ConfigMaps, HPAs, PDBs, Ingresses, and the Prometheus Operator and Istio kinds; rules for an API group that is not installed are inert), plus NetworkPolicies, for remediation; read-only ReplicaSets; leases for leader election.
• No runtime RBAC escalation. The operator never creates or modifies ClusterRoles. It may only bind the pre-provisioned chatcli-watcher and chatcli-role-{viewer,operator,admin,superadmin} (restricted with resourceNames), so a compromised operator cannot bind a more privileged ClusterRole. The chatcli-role-* ClusterRoles are there for you to bind to people; the operator binds nothing to users.
• Remediation guardrails. ApplyManifest only creates or updates the 16 allowed kinds (Deployment, StatefulSet, DaemonSet, Service, ConfigMap, HorizontalPodAutoscaler, PodDisruptionBudget, Ingress, CronJob, Job, ServiceMonitor, PrometheusRule, PodMonitor, ServiceEntry, VirtualService, DestinationRule; extend with security.allowedResourceTypes) in the Issue’s namespace. The operator RBAC (chart and kustomize) grants create and update on each of them, the ServiceMonitor, PodMonitor, PrometheusRule, ServiceEntry, VirtualService and DestinationRule kinds included; a kind you add with security.allowedResourceTypes also needs a ClusterRole rule you add. ReplicaSet is not on the list: its Deployment owns it and would revert a direct write. ExecDiagnostic is limited to the allowlist. Logs pass through 18 scrub patterns (cloud and API keys, JWTs, bearer tokens, passwords, connection strings, private keys, IPv4 addresses, e-mail addresses, long base64/hex secrets) before they reach the LLM.

​

Credentials and rotation

​

Resources

​

High availability

• Operator: run replicaCount: 2 or more with leaderElect: true. The controllers and the WatcherBridge run on the leader only; the REST API and dashboard are served by every replica, so the Service keeps answering during a failover. The chart creates no PodDisruptionBudget for the operator; add one if you drain nodes often.
• Instances: replicas > 1 turns the Service headless and the operator balances round-robin across pods. Each pod is an independent server (its own memory, watcher and emptyDir state), and the alert stream attaches to one pod at a time. For an AIOps Instance, one replica is usually right.
• Persistence and replicas: the sessions PVC is ReadWriteOnce, so with persistence.enabled the Deployment uses the Recreate strategy: the old pod stops and releases the volume before the new one starts, and each rollout (image, configuration or credential change) has a brief downtime. Without persistence the strategy is RollingUpdate. A second replica scheduled on another node still cannot attach the volume (Multi-Attach error): keep one replica with persistence and pin it with scheduling.affinity if your storage is zonal, or run without persistence when you scale out.
• The operator creates no PodDisruptionBudget, HorizontalPodAutoscaler or NetworkPolicy for Instances.

​

Network policies

• Operator: enable networkPolicy.enabled in the chart (or make deploy-network-policy for the raw manifests). Ingress is allowed on the API port (narrow it with networkPolicy.apiIngressFrom), the metrics port (metricsIngressFrom) and the health port. With egress: restricted, egress is limited to DNS, 443, the Kubernetes API port (kubernetesApiPort, 6443), the Instances’ gRPC port (instanceGrpcPort, 50051) and the Prometheus port taken from prometheusUrl; add egressExtraPorts for SMTP (587/465), git over SSH (22) or webhooks on other ports.
• Instances: the operator creates none. A starting point:
  apiVersion: networking.k8s.io/v1
  kind: NetworkPolicy
  metadata:
    name: chatcli
    namespace: chatcli
  spec:
    podSelector:
      matchLabels:
        app.kubernetes.io/name: chatcli
        app.kubernetes.io/instance: chatcli
    policyTypes: [Ingress, Egress]
    ingress:
      - ports: [{port: 50051, protocol: TCP}]   # gRPC: the operator and your clients
        from:
          - namespaceSelector:
              matchLabels:
                kubernetes.io/metadata.name: chatcli-system
      - ports: [{port: 9090, protocol: TCP}]    # metrics + /healthz (kubelet probes, Prometheus)
    egress:
      - ports: [{port: 53, protocol: UDP}, {port: 53, protocol: TCP}]
      - ports: [{port: 443, protocol: TCP}, {port: 6443, protocol: TCP}]   # LLM APIs, Kubernetes API (watcher)
  
  Add a from for your own clients on 50051, and egress to the watcher targets’ metricsPort when you collect their metrics. Kubelet probe traffic is exempt from NetworkPolicy on most CNIs but not all, which is why the example leaves 9090 open to any source.

​

Pod Security

​

Metrics and monitoring

• The server exposes /metrics (and /healthz) on the Instance Service port metrics (9090); the operator exposes controller and AIOps metrics (chatcli_operator_*) on 8080 (serviceMonitor.enabled: true in the chart).
• Both are plain HTTP without authentication, and the server’s metrics listener binds every interface regardless of bindAddress. Restrict them with NetworkPolicy.
• For an Instance, create your own ServiceMonitor:
  apiVersion: monitoring.coreos.com/v1
  kind: ServiceMonitor
  metadata:
    name: chatcli
    namespace: chatcli
  spec:
    selector:
      matchLabels:
        app.kubernetes.io/name: chatcli
        app.kubernetes.io/instance: chatcli
    endpoints:
      - port: metrics
        interval: 30s
  
• Metric names, Grafana dashboards and useful queries: Web Dashboard. Alert on chatcli_operator_instance_ready == 0 and on the ServerReachable condition.

​

Images, versions and upgrades

• Operator and server images are published as ghcr.io/diillson/chatcli-operator:<version> and ghcr.io/diillson/chatcli:<version> (plus latest), multi-arch (amd64, arm64), with SBOM and provenance, signed keyless with cosign:
  cosign verify ghcr.io/diillson/chatcli-operator:1.214.0 \
    --certificate-oidc-issuer https://token.actions.githubusercontent.com \
    --certificate-identity-regexp '^https://github.com/diillson/chatcli/'
  
• Pinning. The chart pins the operator image to its appVersion and passes CHATCLI_OPERATOR_APP_VERSION, so an Instance without spec.image.tag runs the same release and a helm upgrade of the operator upgrades those servers too. Set spec.image.tag only to decouple an Instance from the operator release. Avoid latest in production.
• Upgrading and uninstalling: see Upgrade and uninstall.

​

Backup

• Custom resources, for example kubectl get instances,runbooks,notificationpolicies,escalationpolicies,approvalpolicies,incidentslas,servicelevelobjectives,sourcerepositories,clusterregistrations -A -o yaml; add issues,postmortems,auditevents if you keep incident history in the cluster. Tools such as Velero back up namespaces with their custom resources.
• The Secrets the Instances reference, and chatcli-operator-secrets, from your secret store.
• The sessions PVC holds server sessions. It is owned by the Instance: deleting the Instance deletes the claim (the volume follows its StorageClass reclaim policy).
• LLM spend booked by the pipeline is kept in ConfigMaps chatcli-cost-ledger in the namespaces where Issues occur.

​

What does not exist

• No admission or validating webhook: the CRD schema is the only validation, and the operator reports the rest as conditions.
• No plaintext connection from the operator to an Instance.
• No gRPC reflection unless you set spec.server.security.enableReflection.
• No PodDisruptionBudget, HPA, NetworkPolicy, ServiceMonitor, Ingress or LoadBalancer created for Instances; the Service is always ClusterIP or headless.
• No TLS or authentication on the metrics endpoints.
• No SSO/OIDC for the dashboard: API keys only.
• No file audit log in the operator: its actions are AuditEvent resources. The server has its own audit file (security.auditLogPath).
• No automatic RBAC for people: bind the chatcli-role-* ClusterRoles yourself.

​

Upgrade and uninstall

​

Before you start

1. Back up the custom resources and Secrets (see Backup).
2. Save the values the release runs with, and keep that file as the source of your settings from now on:
   helm get values chatcli-operator -n chatcli-system -o yaml > operator-values.yaml
   
   A release installed without overrides, like the install command on this page, prints null; the file works as it is.
3. Check Crossing these releases for anything that needs action before the upgrade.

​

Run the upgrade

helm upgrade chatcli-operator oci://ghcr.io/diillson/charts/chatcli-operator \
  --version 1.214.0 -n chatcli-system -f operator-values.yaml

git checkout v1.214.0
cd operator
make deploy

​

What happens, in order

1. CRDs. The pre-upgrade hook Job re-applies the 17 CRDs (kubectl apply --server-side) before anything else changes.
2. Operator. The operator Deployment rolls: the new pod becomes Ready before the old one stops, so the dashboard and REST API stay up. The controllers restart on the new pod.
3. Instances. The new operator re-renders every Instance. An Instance without spec.image.tag gets the new server image, and any change to the pod template rolls its pods. With persistence.enabled the strategy is Recreate: the old pods stop before the new ones start, so each such Instance has a short outage.
4. Probe. The operator probes each server again. A probe that lands inside the rollout can record ServerReachable=False / ProbeFailed with DeadlineExceeded ... while waiting for connections to become ready, connection refused or i/o timeout, even though the new pods serve fine a moment later. A failed probe is retried after 30 seconds, so the condition turns True shortly after the rollout.
5. Alerts. The WatcherBridge reconnects. If it reaches a server older than 1.211.0, which has no StreamAlerts (typically an old pod that is still running), it logs Server has no StreamAlerts RPC; polling GetAlerts and polls every 30 seconds. When that old pod goes away, the next poll fails with Unavailable, the bridge logs Server unavailable while polling; retrying the alert stream on reconnect and opens the stream on the new server. No alert is lost meanwhile.

​

Verify

helm -n chatcli-system list                                    # CHART chatcli-operator-1.214.0
kubectl get crd -o name | grep -c 'platform.chatcli.io'        # 17
kubectl -n chatcli-system rollout status deploy/chatcli-operator
kubectl get instances -A                                       # READY true, VERSION v1.214.0
kubectl -n chatcli get instance chatcli \
  -o jsonpath='{range .status.conditions[*]}{.type}={.status} {.reason}{"\n"}{end}'

​

Crossing these releases

​

Uninstall

​

Troubleshooting

​

Development

cd operator

make build                 # fmt, vet, go build -o bin/manager main.go
make test                  # unit tests (fake client)
make test-integration      # envtest: real kube-apiserver + etcd (Kubernetes 1.37.0), the CRDs from
                           # config/crd/bases and the controllers wired as main.go wires them;
                           # binaries fetched into ./bin; skipped when KUBEBUILDER_ASSETS is unset
make dash-preview          # the web dashboard over synthetic data, no cluster:
                           # http://127.0.0.1:8085/, API key "preview" (DASHPREVIEW_ADDR overrides)
make run                   # run the operator locally against your current kubeconfig

# Images (the build context is the repository root)
make docker-build IMG=registry.example.com/chatcli-operator:dev
make docker-push  IMG=registry.example.com/chatcli-operator:dev
# Server image; VERSION is what the Instance's VERSION column reports
docker build -t registry.example.com/chatcli:dev \
  --build-arg VERSION=dev-$(git rev-parse --short HEAD) ..

# Deploy your build
make deploy IMG=registry.example.com/chatcli-operator:dev
# or
helm install chatcli-operator ../deploy/helm/chatcli-operator -n chatcli-system --create-namespace \
  --set image.repository=registry.example.com/chatcli-operator --set image.tag=dev

​

Next steps