Production Security Hardening

This guide hardens the two production shapes:

the ChatCLI server, installed with the chatcli Helm chart or managed by the operator as an Instance;
the operator, with its REST API and web dashboard.

It only uses controls that exist in the code. The last section lists what ChatCLI does not protect, and what to put in place instead.

For the full AIOps install (operator, Instance, policies, observability), follow AIOps Production Setup. This page is about the security settings on top of it.

Prerequisites

Helm 3.8 or later (Helm 4 included), kubectl, openssl.
A namespace for the server (chatcli below) and rights to create Secrets in it.
For the certificate steps, cert-manager or your own CA.

The server’s security model

Read this before you choose settings. It explains what each control does and does not cover.

Area	Behavior
Listener	gRPC on port 50051. Inside Kubernetes it binds every interface, and refuses to start without a credential (token, JWT material or client CA). JWT material that fails to load does not count: if it is the only credential, the server refuses to start.
TLS	Optional on the server, TLS 1.3 minimum when on. The operator always dials TLS, so an operator-managed Instance must enable it.
Shared token	Every holder is the same identity (`legacy-token`) with the admin role, and they all share one rate-limit bucket.
JWT	HS256 (`CHATCLI_JWT_SECRET`) or RS256 (`CHATCLI_JWT_PUBLIC_KEY`), never both. `exp` is required; `iss` and `aud` are checked when configured. The server verifies tokens but never issues them.
mTLS	`CHATCLI_SERVER_TLS_CLIENT_CA` makes every connection present a client certificate. A caller with only a certificate gets `CHATCLI_MTLS_ROLE` (default `user`). A bearer token, when present, wins over the certificate.
Roles	`admin`, `user` (JWT `operator` or `user`) and `readonly` (JWT `viewer` or `readonly`). Roles are enforced only on the pipeline execution RPCs (admin), some hub operations, plugin management and remote plugins (a read-only caller lists and downloads none). A read-only caller can still send prompts and manage its sessions, so treat every valid credential as permission to spend your LLM budget.
Rate limit	Per authenticated subject, 10 requests per second with a burst of 20 by default. It is always on.
Unauthenticated endpoints	`grpc.health.v1` and the `Health` RPC; the metrics port (`/metrics`, `/healthz`), plain HTTP on every interface.

Step 1: Choose and create the credential

Shared token
JWT HS256
JWT RS256
mTLS

The simplest option, for machine clients you control. Everyone who has it is admin.

kubectl create namespace chatcli
kubectl -n chatcli create secret generic chatcli-token \
  --from-literal=token="$(openssl rand -hex 32)"

Server chart: put CHATCLI_SERVER_TOKEN in the Secret named by secrets.existingSecret, or set server.token, which the chart stores in a Secret and never passes as a command-line argument. Instance: spec.server.token: {name: chatcli-token, key: token}.

One shared secret verifies tokens you mint, each with its own subject, role and expiry.

kubectl -n chatcli create secret generic chatcli-jwt \
  --from-literal=secret="$(openssl rand -hex 32)"

Server chart: security.jwtSecretRef: {name: chatcli-jwt, key: secret}, plus security.jwtIssuer and security.jwtAudience (leave the inline security.jwtSecret empty: setting both fails the render). Instance: spec.server.security.jwtSecretRef, jwtIssuer, jwtAudience; the operator then mints its own short-lived tokens with the same secret.The server does not issue tokens. Mint them with your identity system, or, for a first test, with this script (standard library only):

export JWT_SECRET="$(kubectl -n chatcli get secret chatcli-jwt -o jsonpath='{.data.secret}' | base64 -d)"
python3 - <<'EOF'
import base64, hashlib, hmac, json, os, time
b64 = lambda d: base64.urlsafe_b64encode(d).rstrip(b"=")
header = b64(json.dumps({"alg": "HS256", "typ": "JWT"}).encode())
claims = b64(json.dumps({
    "sub": "alice@example.com",
    "role": "user",                  # admin | operator/user | viewer/readonly
    "iss": "chatcli-prod",           # must equal security.jwtIssuer
    "aud": "chatcli",                # must equal security.jwtAudience (a string, not a list)
    "exp": int(time.time()) + 3600,  # required
}).encode())
sig = b64(hmac.new(os.environ["JWT_SECRET"].encode(), header + b"." + claims, hashlib.sha256).digest())
print((header + b"." + claims + b"." + sig).decode())
EOF

Your identity provider signs, and the server holds only the public key.

kubectl -n chatcli create secret generic chatcli-jwt-pub \
  --from-file=public.pem=idp-public.pem

Server chart: security.jwtPublicKeyRef: {name: chatcli-jwt-pub, key: public.pem} (not together with the inline security.jwtPublicKey: both fail the render). Instance: spec.server.security.jwtPublicKeyRef. The operator cannot mint RS256 tokens, so it needs its own credential: spec.server.security.operatorTokenRef (a token your IdP issued for it) or operatorClientCertSecretName.The aud claim is compared as a single string. Tokens whose aud is a list are rejected when jwtAudience is set.

Every connection must present a certificate signed by your client CA. Use a CA dedicated to clients, not the CA that signed the server certificate, or every certificate that CA ever issued becomes a valid client.

kubectl -n chatcli create secret generic chatcli-client-ca \
  --from-file=ca.crt=client-ca.crt

Server chart: mount it with extraVolumes/extraVolumeMounts and set security.tlsClientCA to the mounted path (TLS must be on). Instance: spec.server.tls.clientCASecretName: chatcli-client-ca, plus spec.server.security.operatorClientCertSecretName for the operator’s own certificate.Clients: CHATCLI_TLS_CLIENT_CERT and CHATCLI_TLS_CLIENT_KEY with chatcli connect --tls. There is no command-line flag for the client certificate.

Step 2: TLS certificates

The certificate must cover the name clients dial. An operator-managed Instance is dialed as <instance>.<namespace>.svc.cluster.local, which needs a private CA; see TLS certificate in the AIOps setup. For a server reached from outside the cluster under a public name:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: chatcli-tls
  namespace: chatcli
spec:
  secretName: chatcli-tls
  issuerRef:
    name: letsencrypt-prod        # your ClusterIssuer
    kind: ClusterIssuer
  dnsNames:
    - chatcli.example.com

With the server chart, TLS needs two values. tls.existingSecret mounts the Secret at /etc/chatcli/tls, and the certificate and key paths default to /etc/chatcli/tls/tls.crt and /etc/chatcli/tls/tls.key (set certFile / keyFile only for other file names). tls.enabled: true without a Secret or both paths fails the install rather than serving plaintext:

tls:
  enabled: true
  existingSecret: chatcli-tls

cert-manager renewals change the Secret, but the server reads the certificate only at startup. Restart the Deployment after a renewal, or use an operator-managed Instance, which rolls its pods when the TLS Secret changes.

Step 3: Deploy the server with hardened values

Every key below exists in the chart’s schema, which rejects unknown keys. This example uses HS256 JWTs.

kubectl -n chatcli create secret generic chatcli-llm-keys \
  --from-literal=ANTHROPIC_API_KEY='<anthropic-key>'
kubectl -n chatcli create secret generic chatcli-encryption-key \
  --from-literal=key="$(openssl rand -hex 32)"

# values-prod.yaml
llm:
  provider: CLAUDEAI
  model: claude-sonnet-5

secrets:
  existingSecret: chatcli-llm-keys

server:
  port: 50051
  metricsPort: 9090             # /metrics and /healthz, plain HTTP: keep it behind the NetworkPolicy
  grpcReflection: false

tls:
  enabled: true
  existingSecret: chatcli-tls   # certFile/keyFile default to /etc/chatcli/tls/tls.crt and tls.key

security:
  jwtSecretRef:
    name: chatcli-jwt
    key: secret
  jwtIssuer: chatcli-prod
  jwtAudience: chatcli
  rateLimitRps: 20
  rateLimitBurst: 40
  auditLogPath: /var/log/chatcli/audit.jsonl
  agentSecurityMode: strict     # the default; set it so a stray env var cannot flip it
  envRedactMode: strict

extraEnv:
  - name: CHATCLI_ENCRYPTION_KEY
    valueFrom:
      secretKeyRef:
        name: chatcli-encryption-key
        key: key

extraVolumes:
  - name: audit
    persistentVolumeClaim:
      claimName: chatcli-audit   # create it first; the root filesystem is read-only
extraVolumeMounts:
  - name: audit
    mountPath: /var/log/chatcli

persistence:
  enabled: true
  size: 5Gi

rbac:
  create: false                  # no Secret access either way, but it grants workload and node remediation verbs; skip it when the watcher is off

networkPolicy:
  enabled: true
  ingressFrom:                   # applies to the gRPC and metrics ports
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: chatcli-clients
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: monitoring
  egress: restricted             # DNS, 443 and the Kubernetes API only
  kubernetesApiPort: 6443

resources:
  requests:
    cpu: 200m
    memory: 256Mi
  limits:
    cpu: "1"
    memory: 1Gi

serviceMonitor:
  enabled: true

helm install chatcli oci://ghcr.io/diillson/charts/chatcli \
  --version 1.226.0 \
  --namespace chatcli \
  -f values-prod.yaml

The chart’s default pod settings already run as non-root (UID 1000) with a read-only root filesystem, no capabilities and the RuntimeDefault seccomp profile, and meet the restricted Pod Security Standard. Probes use /healthz on the metrics port and a TCP check on the gRPC port, which work with TLS on.

The chart renders a PodDisruptionBudget only when replicaCount is greater than 1. Several replicas on one ReadWriteOnce sessions volume fail to schedule across nodes. Use persistence.accessModes: [ReadWriteMany] with a storage class that supports it, or stay on one replica; ReadWriteOncePod with more than one replica fails the render. Without ReadWriteMany, the chart stops the old pod before starting the new one on each rollout (maxSurge: 0, maxUnavailable: 1), so an upgrade has a short gap instead of hanging on the volume; see Rollouts on the sessions volume.

The same controls on an operator-managed Instance:

spec:
  server:
    token: {name: chatcli-token, key: token}   # or security.jwtSecretRef / jwtPublicKeyRef
    tls:
      enabled: true
      secretName: chatcli-prod-tls
      # clientCASecretName: chatcli-client-ca   # mTLS
    security:
      rateLimitRps: 20
      rateLimitBurst: 40
      # enableReflection stays unset (false)
  features:
    encryptionKeyRef: {name: chatcli-encryption-key, key: key}
  resources:                                    # the operator sets none
    requests: {cpu: 250m, memory: 512Mi}
    limits: {cpu: "1", memory: 1Gi}

Step 4: Audit trail

With CHATCLI_AUDIT_LOG_PATH (chart security.auditLogPath, Instance spec.server.security.auditLogPath), the server appends one JSON line per RPC to a hash-chained file. Each line records the time, action and method, the caller (user:<subject> or anonymous), the role, the client host, the result (success, error or denied) and the duration. The path must be absolute and on a writable, persistent volume, as in step 3. On an Instance the only writable paths are emptyDir volumes, lost when the pod restarts, unless you enable persistence. To check that nobody edited the file, read it from the volume (for example in a helper pod that mounts the same PVC) and run, in the chatcli REPL:

/config security verify-audit /path/to/audit.jsonl

The operator does not write a file audit log. It records its actions as AuditEvent resources, readable with kubectl get auditevents -A and exportable through GET /api/v1/audit/export. Enable the Kubernetes API server audit log to capture who changed ChatCLI resources.

Step 5: Encryption at rest

The server encrypts what it writes under ~/.chatcli (sessions, history, memory and other local stores), and on Kubernetes this is on by default with a key in a Secret the server or the operator creates. For production, supply the key yourself so it lives in your secret manager: security.encryptionKeySecret in the chart, spec.features.encryptionKeyRef on an Instance, or a Secret as in step 3. Do not use the chart’s security.encryptionKey: it puts the key in the pod spec as a plain value. Back the key Secret up with the PVC data; see Encryption at Rest. Rotation is described in Security.

Step 6: Agent, pipeline and plugin controls

These matter only if the server runs agent work or loads plugins.

Pipeline. pipeline.enabled (Instance spec.pipeline.enabled) lets clients run the coder, agent and tools on the server. Those RPCs require the admin role. Leave it off unless you need it.
Command allowlist. strict is the default mode: only commands in the allowlist run, and CHATCLI_AGENT_ALLOWLIST adds more (separated by , or ;). The built-in list includes shells and interpreters such as bash, python, node and eval, so it is not a sandbox. Contain the pod instead: restricted NetworkPolicy egress, no chart RBAC when the watcher is off (rbac.create: false), and nothing sensitive on the pod’s volumes.
Plugins. At load, every tool plugin and provider plugin needs a detached signature (<plugin>.sig) that verifies against a trusted public key — in ~/.chatcli/trusted-keys/, or for a fleet in the managed /etc/chatcli/trusted-keys/ that every user trusts. Unsigned plugins, and plugins on a machine with no trusted key, are skipped unless CHATCLI_ALLOW_UNSIGNED_PLUGINS=true, which you must never set in production. A plugin whose signature does not verify is always skipped.
chatcli plugin keygen --output ./keys # plugin-signing.key and plugin-signing.pub chatcli plugin sign --binary ./my-plugin --key ./keys/plugin-signing.key chatcli plugin trust --key ./keys/plugin-signing.pub # on each machine that loads it chatcli plugin verify --binary ./my-plugin
In a pod, ~/.chatcli is /home/chatcli/.chatcli. Mount the trusted public keys at /home/chatcli/.chatcli/trusted-keys with extraVolumes/extraVolumeMounts. Keep the private key out of the cluster.

Step 7: Operator, REST API and dashboard

API keys

The REST API and dashboard authenticate only with the X-API-Key header, against the Secret chatcli-operator-secrets (key api-keys) in the operator’s namespace. Create one key per team and role (viewer, operator, admin), and keep admin keys few. See Dashboard and REST API keys for the commands.Keys are re-read every 30 seconds. To revoke one, remove it from the Secret. Deleting both the Secret and the ConfigMap chatcli-operator-config revokes every key within about 30 seconds. An edit that leaves invalid YAML does not revoke anything: the last valid key set stays in force and the operator logs the error. Give each approver their own key and set its name: approvals are recorded against the key, and a quorum counts each key once.

Never enable dev mode

security.devMode: true (CHATCLI_OPERATOR_DEV_MODE) makes the REST API accept every caller as admin while no keys are loaded. The variable is read with boolean semantics, case-insensitively: true, TRUE, 1 and t all turn it on, and the startup log reports the same mode the API applies. Keep it false.

TLS for the REST API

Terminate TLS at your ingress, or serve HTTPS (TLS 1.3) from the operator itself:

# values-operator.yaml
extraVolumes:
  - name: api-tls
    secret:
      secretName: chatcli-operator-api-tls
extraVolumeMounts:
  - name: api-tls
    mountPath: /etc/chatcli-operator/api-tls
    readOnly: true
security:
  apiTLS:
    certFile: /etc/chatcli-operator/api-tls/tls.crt
    keyFile: /etc/chatcli-operator/api-tls/tls.key

CORS stays closed

Cross-origin calls are denied until you name an origin in security.corsAllowedOrigins. The dashboard is served from the same origin as the API and needs none.

NetworkPolicy

Set networkPolicy.enabled: true on the operator chart, restrict apiIngressFrom and metricsIngressFrom, and keep egress: restricted. The values are in step 1 of the AIOps setup.

RBAC on ChatCLI resources

The operator runs with a cluster-wide ClusterRole that can read and write Secrets in every namespace, change workloads, create pods (chaos stress pods), evict pods (DrainNode) and update nodes. No admission webhook checks what users submit, so Kubernetes RBAC is the only gate on who can make the operator act. Grant create/update on remediationplans, runbooks, instances, chaosexperiments, approvalrequests and the policy kinds only to the people who should hold that power. The chart pre-provisions the ClusterRoles chatcli-role-viewer, chatcli-role-operator, chatcli-role-admin and chatcli-role-superadmin; bind them yourself, since nothing binds them automatically.

Remediation guardrails

security.allowedDiagnosticCommands adds commands to the read-only ExecDiagnostic allowlist; everything else is refused.
security.allowedResourceTypes adds kinds that the ApplyManifest action may create. It extends the built-in list and never narrows it.
security.logScrubPatterns adds comma-separated regular expressions to the scrubbing applied to the logs and context the operator sends to the LLM. It does not scrub the operator’s own logs.
Gate risky actions with an ApprovalPolicy: see Approvals.

Step 8: Verify the images

Release images are signed with cosign (keyless) and carry SBOM and provenance attestations.

cosign verify ghcr.io/diillson/chatcli:1.226.0 \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com \
  --certificate-identity-regexp '^https://github\.com/diillson/chatcli/\.github/workflows/(3-publish-release|image-refresh)\.yml@refs/heads/main$'

Use the same command for ghcr.io/diillson/chatcli-operator:1.226.0. Pin image tags, or digests, in production instead of latest.

Step 9: Verify the hardening

TLS 1.3 only, with the right name

kubectl -n chatcli port-forward svc/chatcli 50051:50051 &
openssl s_client -connect localhost:50051 -servername chatcli.example.com -tls1_3 </dev/null 2>/dev/null \
  | openssl x509 -noout -subject -ext subjectAltName
openssl s_client -connect localhost:50051 -tls1_2 </dev/null   # handshake fails

Health answers, without a credential

kubectl -n chatcli exec deploy/chatcli -- chatcli healthcheck   # localhost:50051 is serving.

The check reads the server’s own TLS settings, flags included, and verifies the server against the certificate it serves. With mTLS, set CHATCLI_TLS_CLIENT_CERT and CHATCLI_TLS_CLIENT_KEY in the pod.

Calls without a credential are refused

chatcli connect --addr localhost:50051 --tls --ca-cert ca.crt -p "hello"
# fails: authentication failed
chatcli connect --addr localhost:50051 --tls --ca-cert ca.crt --token "$JWT" -p "hello"
# answers

ca.crt is the CA of the server certificate (not needed for a public CA). $JWT is a token from step 1.

Reflection is off

grpcurl -cacert ca.crt localhost:50051 list
# Failed to list services: server does not support the reflection API

Pod settings

kubectl -n chatcli get pod -l app.kubernetes.io/name=chatcli \
  -o jsonpath='{.items[0].spec.securityContext}{"\n"}{.items[0].spec.containers[0].securityContext}{"\n"}'

Network isolation

kubectl -n chatcli get networkpolicy
# From a namespace that is not allowed, both ports must time out:
kubectl -n default run np-test --rm -it --restart=Never --image=curlimages/curl -- \
  curl -m 5 http://chatcli.chatcli.svc:9090/metrics

Audit lines are written

After a few calls, the audit file on the volume grows by one line per RPC, and /config security verify-audit reports the chain as intact.

Operator REST API

kubectl -n chatcli-system port-forward svc/chatcli-operator 8090:8090 &
curl -s -o /dev/null -w '%{http_code}\n' http://localhost:8090/api/v1/incidents   # 401

What ChatCLI does not protect

These gaps exist in current releases. Cover each one with the mitigation next to it.

Gap	What it means	Mitigation
No admission webhook	Nothing validates ChatCLI resources before they are stored. Anyone who can create a RemediationPlan, Runbook, Instance or ChaosExperiment can make the operator act with its ClusterRole	Kubernetes RBAC on the `platform.chatcli.io` resources; ApprovalPolicies for remediation
Metrics endpoints without authentication	The server’s metrics port (`/metrics`, `/healthz`) listens on every interface in plain HTTP, and so does the operator’s port 8080	NetworkPolicy allowing only Prometheus (and the node CIDR for probes, where your CNI needs it)
No SSO or OIDC for the dashboard	Access is by API key only. The browser keeps the key in `localStorage`. Dashboard approvals are recorded as the typed name plus the key’s identity: the name is free text, only the key is authenticated	Few keys, one per team and role, and one per approver where quorum rules apply (set each key’s `name`); TLS; restrict ingress sources; put an authenticating proxy in front if you need per-person login
Coarse server roles	A `viewer`/`readonly` credential can still send prompts and manage its sessions	Issue credentials only to people allowed to use the LLM; run separate servers per trust boundary
Shared token is admin	Every holder has the same admin identity and shares one rate-limit bucket	JWT or mTLS for people; the token only for machine clients
No operator file audit log	The operator’s own actions are only in `AuditEvent` resources	Kubernetes API audit logging; export `AuditEvent`s through the REST API
No automatic user RBAC	The `chatcli-role-*` ClusterRoles are never bound for you	Bind them to your groups
No NetworkPolicy, PDB or HPA for Instances	The operator creates none of them	Create them yourself (see the AIOps setup)
Optional Secret references on Instances	A missing Secret or key silently drops the variable; with the token gone the pod exits on startup	Check the Instance conditions and pod status after every change
Server log destinations	In a pod the log goes to stderr (`kubectl logs`) and to the pod’s ephemeral file; the file is lost with the pod	Ship `kubectl logs` through your log collector; the audit trail is the durable record
Authentication failure limiter	Each client host may fail authentication 5 times in a burst, then once every 12 seconds; a host that exhausts it is refused (`authentication failed`, log `auth failure rate limit exceeded`) until a slot refills. Valid credentials never spend from it	Nothing to tune; find the caller with the wrong or expired credential

Final checklist

Item	Done
Credential chosen: JWT or mTLS for people, shared token only for machines
JWT `iss` and `aud` configured, tokens short-lived
TLS on, certificate covering the dialed name, renewal handled
mTLS client CA separate from the server CA (if used)
Reflection off (`grpcReflection` / `enableReflection` false)
Audit log on a persistent volume, chain verified
Encryption key from a Secret, not a plain value
Pipeline off unless needed; plugins signed and trusted keys mounted
Server NetworkPolicy with restricted ingress and egress
Server chart RBAC off when the watcher is off
Operator dev mode off, API keys per team, revocation by editing the Secret
REST API behind TLS, CORS closed
Operator NetworkPolicy on
RBAC on `platform.chatcli.io` resources limited to the right people
Images verified with cosign and pinned

Next steps

Security

Every security control in detail.

AIOps Production Setup

The complete operator and Instance runbook.

K8s Operator

Instance fields and what the operator creates.

Environment variables

Every variable the server and the operator read.

Get started

Documentation

Production Security Hardening

Prerequisites

The server’s security model

Step 1: Choose and create the credential

Step 2: TLS certificates

Step 3: Deploy the server with hardened values

Step 4: Audit trail

Step 5: Encryption at rest

Step 6: Agent, pipeline and plugin controls

Step 7: Operator, REST API and dashboard

Step 8: Verify the images

Step 9: Verify the hardening

What ChatCLI does not protect

Final checklist

Next steps

Security

AIOps Production Setup

K8s Operator

Environment variables

​Prerequisites

​The server’s security model

​Step 1: Choose and create the credential

​Step 2: TLS certificates

​Step 3: Deploy the server with hardened values

​Step 4: Audit trail

​Step 5: Encryption at rest

​Step 6: Agent, pipeline and plugin controls

​Step 7: Operator, REST API and dashboard

​Step 8: Verify the images

​Step 9: Verify the hardening

​What ChatCLI does not protect

​Final checklist

​Next steps

Security

AIOps Production Setup

K8s Operator

Environment variables

Prerequisites

The server’s security model

Step 1: Choose and create the credential

Step 2: TLS certificates

Step 3: Deploy the server with hardened values

Step 4: Audit trail

Step 5: Encryption at rest

Step 6: Agent, pipeline and plugin controls

Step 7: Operator, REST API and dashboard

Step 8: Verify the images

Step 9: Verify the hardening

What ChatCLI does not protect

Final checklist

Next steps