Production Security Hardening

This guide shows how to deploy ChatCLI in production with all security measures enabled. Follow each step in order to ensure a complete configuration.

Prerequisites

Kubernetes cluster with Helm 3.8+
kubectl and helm configured
openssl available for key generation
Access to create Secrets in the target namespace

Step 1: Generate JWT Secret and Create K8s Secret

JWT is used to authenticate connections between clients and the ChatCLI server.

# Generate a 256-bit JWT secret
JWT_SECRET=$(openssl rand -hex 32)

# Create the Secret in Kubernetes
kubectl create namespace chatcli
kubectl -n chatcli create secret generic chatcli-jwt \
  --from-literal=secret="$JWT_SECRET"

Store the $JWT_SECRET value in a secrets vault (Vault, AWS Secrets Manager, etc.). You will need it to configure remote clients.

Verify the Secret:

kubectl -n chatcli get secret chatcli-jwt -o jsonpath='{.data.secret}' | base64 -d

Step 2: Configure TLS Certificates

cert-manager (recommended)
Manual certificate

# chatcli-certificate.yaml
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: chatcli-tls
  namespace: chatcli
spec:
  secretName: chatcli-tls-certs
  issuerRef:
    name: letsencrypt-prod
    kind: ClusterIssuer
  dnsNames:
    - chatcli.mydomain.com

kubectl apply -f chatcli-certificate.yaml

# Generate self-signed certificate (for testing only)
openssl req -x509 -newkey rsa:4096 -keyout tls.key -out tls.crt \
  -days 365 -nodes -subj "/CN=chatcli.mydomain.com"

kubectl -n chatcli create secret tls chatcli-tls-certs \
  --cert=tls.crt --key=tls.key

The snippet above covers the standalone chatcli server exposed on a public domain. If you are using the Operator + Instance CR to dial gRPC via in-cluster DNS (<instance>.<ns>.svc.cluster.local), this cert will fail with x509: certificate is not valid for any names (no SANs) and x509: certificate signed by unknown authority (no ca.crt in the Secret). For that path, use the openssl.cnf with subjectAltName and the 3-key Secret documented in AIOps Production Setup §2.1.

Step 3: Set Up Rate Limiting

Define request limits to prevent abuse and DoS:

# values-security.yaml (partial)
security:
  rateLimitRps: 20        # 20 requests per second
  # bindAddress: "0.0.0.0"  # Optional — auto-detected in Kubernetes

In Kubernetes, bindAddress is automatically set to 0.0.0.0 via KUBERNETES_SERVICE_HOST detection. Only set it explicitly for non-Kubernetes server deployments.

For multi-tenant environments, consider lower values (5-10 RPS per instance) and use HPA to scale horizontally.

Step 4: Enable Audit Logging

Audit logging records each operation with details of who, what, and when:

# values-security.yaml (partial)
security:
  auditLog: true

# Equivalent environment variable
export CHATCLI_AUDIT_LOG=true

Audit logs include:

Authenticated user (via JWT claims)
Command or operation executed
Timestamp and result (success/failure)
Source IP of the request

Step 5: Configure Agent Command Allowlist

Enable strict mode to ensure only approved commands are executed:

# values-security.yaml (partial)
security:
  agentSecurityMode: strict

# If you need additional commands:
env:
  - name: CHATCLI_AGENT_ALLOWLIST
    value: "terraform;ansible;packer;vault"
  - name: CHATCLI_AGENT_WORKSPACE_STRICT
    value: "true"
  - name: CHATCLI_MAX_COMMAND_OUTPUT
    value: "50000"

In strict mode, over 150 common commands are already pre-approved across 8 categories (file, text, dev, containers, network, system, editors, shell). Only add commands specific to your workflow.

Step 6: Sign Plugins with Ed25519

To ensure only trusted plugins are loaded:

# Generate Ed25519 key pair
openssl genpkey -algorithm Ed25519 -out plugin-sign.key
openssl pkey -in plugin-sign.key -pubout -out plugin-sign.pub

# Extract the public key in base64 format
PLUGIN_PUB_KEY=$(openssl pkey -in plugin-sign.key -pubout -outform DER | base64)

# Sign a plugin
openssl pkeyutl -sign -inkey plugin-sign.key \
  -rawin -in my-plugin.so -out my-plugin.so.sig

Configure ChatCLI to verify signatures:

export CHATCLI_PLUGIN_VERIFY_SIGNATURES=true
export CHATCLI_PLUGIN_TRUSTED_KEYS="$PLUGIN_PUB_KEY"

Step 7: Set Up Session Encryption

Enable encryption for sessions stored on disk:

# Generate AES-256 encryption key
SESSION_KEY=$(openssl rand -hex 32)

kubectl -n chatcli create secret generic chatcli-session-key \
  --from-literal=key="$SESSION_KEY"

# values-security.yaml (partial)
security:
  sessionEncryption: true
env:
  - name: CHATCLI_SESSION_ENCRYPTION_KEY
    valueFrom:
      secretKeyRef:
        name: chatcli-session-key
        key: key

Step 8: Configure Operator Security

For environments with the K8s operator, configure additional protections. First, create a Secret with the operator API keys (dashboard / REST API auth):

apiVersion: v1
kind: Secret
metadata:
  name: chatcli-operator-secrets
  # Must match the operator pod's namespace (POD_NAMESPACE / SA file).
  # Code default: chatcli-system. Adjust if you ran `helm install --namespace <other>`.
  namespace: chatcli-system
type: Opaque
stringData:
  api-keys: |
    - key: "<your-api-key>"
      role: admin
      description: "Dashboard admin"

This Secret is different from the chatcli-api-keys Secret consumed by the chatcli server (which carries OPENAI_API_KEY, ANTHROPIC_API_KEY etc. via Instance.spec.apiKeys.name). See Security — Operator Authentication for the comparison table.

Changes to the Secret chatcli-operator-secrets (or the ConfigMap chatcli-operator-config as fallback — same api-keys field) are picked up automatically within 30 seconds. No operator restart is needed.

Then, configure the security environment variables:

# values-security.yaml (partial)
env:
  - name: CHATCLI_OPERATOR_FAIL_CLOSED
    value: "true"
  - name: CHATCLI_OPERATOR_RESOURCE_ALLOWLIST
    value: "deployments;services;configmaps;pods"
  - name: CHATCLI_OPERATOR_LOG_SCRUBBING
    value: "true"

Setting	Effect
`FAIL_CLOSED`	Blocks operations when the agent is unavailable (vs. allowing insecure fallback)
`RESOURCE_ALLOWLIST`	Limits which K8s resources the operator can manipulate
`LOG_SCRUBBING`	Removes tokens, passwords, and sensitive data from logs

Step 9: Deploy with Helm (Complete Configuration)

Combine all configurations into a single values-prod.yaml:

# values-prod.yaml
llm:
  provider: CLAUDEAI
secrets:
  existingSecret: chatcli-llm-keys
server:
  token: ""  # Using JWT via jwtSecretRef
  metricsPort: 9090
tls:
  enabled: true
  existingSecret: chatcli-tls-certs
security:
  rateLimitRps: 20
  # bindAddress: "0.0.0.0"  # Optional — auto-detected in Kubernetes
  agentSecurityMode: strict
  auditLog: true
  sessionEncryption: true
  jwtSecretRef:
    name: chatcli-jwt
    key: secret
persistence:
  enabled: true
  size: 5Gi
resources:
  requests:
    memory: 256Mi
    cpu: 200m
  limits:
    memory: 1Gi
    cpu: "1"
podSecurityContext:
  runAsNonRoot: true
  runAsUser: 1000
  runAsGroup: 1000
  fsGroup: 1000
  seccompProfile:
    type: RuntimeDefault
securityContext:
  allowPrivilegeEscalation: false
  readOnlyRootFilesystem: true
  capabilities:
    drop:
      - ALL
networkPolicy:
  enabled: true
podDisruptionBudget:
  enabled: true
  minAvailable: 1
serviceMonitor:
  enabled: true
  interval: 30s

helm install chatcli oci://ghcr.io/diillson/charts/chatcli \
  --namespace chatcli --create-namespace \
  -f values-prod.yaml

Step 10: Verify Security

Run this checklist to confirm everything is configured correctly:

Verify TLS

kubectl -n chatcli get secret chatcli-tls-certs
# Should return the secret with type: kubernetes.io/tls

Verify JWT Secret

kubectl -n chatcli get secret chatcli-jwt
# Should exist with the 'secret' key

Verify Pod SecurityContext

kubectl -n chatcli get pod -l app.kubernetes.io/name=chatcli -o jsonpath='{.items[0].spec.securityContext}'
# Should show runAsNonRoot: true, etc.

Verify NetworkPolicy

kubectl -n chatcli get networkpolicy
# Should list the ChatCLI NetworkPolicy

Verify gRPC Reflection is disabled

kubectl -n chatcli exec deploy/chatcli -- env | grep GRPC_REFLECTION
# Should be empty or 'false'

Test authenticated connection

chatcli connect chatcli.mydomain.com:50051 --token "$JWT_SECRET" --tls

Verify rate limiting

# Send multiple rapid requests -- should return 429 error after the limit
for i in $(seq 1 30); do
  chatcli -p "ping" --remote chatcli.mydomain.com:50051 &
done

Verify audit logs

kubectl -n chatcli logs deploy/chatcli | grep "audit"
# Should show audit entries for the operations above

Final Checklist

Item	Status
JWT Secret created and referenced
TLS enabled with valid certificate
Rate limiting configured
Audit logging enabled
Agent `strict` mode active
Workspace strict enabled
Plugins with signature verification
Session encryption active
Operator in fail-closed mode
Log scrubbing enabled
NetworkPolicy created
PodDisruptionBudget created
Restrictive SecurityContext
gRPC Reflection disabled
ServiceMonitor for metrics

Next Steps

Security and Hardening

Complete documentation of all security measures.

Deploy with Docker and Helm

Complete containerized deployment guide.

K8s Operator

Configure the Kubernetes operator.

Configuration Reference

All available environment variables.

​Prerequisites

​Step 1: Generate JWT Secret and Create K8s Secret

​Step 2: Configure TLS Certificates

​Step 3: Set Up Rate Limiting

​Step 4: Enable Audit Logging

​Step 5: Configure Agent Command Allowlist

​Step 6: Sign Plugins with Ed25519

​Step 7: Set Up Session Encryption

​Step 8: Configure Operator Security

​Step 9: Deploy with Helm (Complete Configuration)

​Step 10: Verify Security

​Final Checklist

​Next Steps

Security and Hardening

Deploy with Docker and Helm

K8s Operator

Configuration Reference

Prerequisites

Step 1: Generate JWT Secret and Create K8s Secret

Step 2: Configure TLS Certificates

Step 3: Set Up Rate Limiting

Step 4: Enable Audit Logging

Step 5: Configure Agent Command Allowlist

Step 6: Sign Plugins with Ed25519

Step 7: Set Up Session Encryption

Step 8: Configure Operator Security

Step 9: Deploy with Helm (Complete Configuration)

Step 10: Verify Security

Final Checklist

Next Steps