Incidents
Detection, ack, snooze, timeline, remediation, and resolution
Runbooks
Full CRUD for runbooks
Analytics
MTTD, MTTR, trends, top resources, capacity, compliance, cost
SLOs
Targets, error budget, and burn rate
Federation
Multi-cluster status, cross-cluster correlations
Health
Liveness and readiness endpoints
Base URL
/) listens on the operator pod, port 8090, exposed as port api of the Service chatcli-operator in the operator namespace (chatcli-system by default). It is not served by the Service of an Instance (that one is the gRPC server on 50051).
The port comes from the chart value
api.port (default 8090), which sets the CHATCLI_AIOPS_PORT environment variable of the operator. To serve HTTPS (TLS 1.3 minimum), set CHATCLI_AIOPS_TLS_CERT and CHATCLI_AIOPS_TLS_KEY to certificate and key files inside the operator container (chart: security.apiTLS.certFile / security.apiTLS.keyFile). The operator chart has no extraVolumes value, so the files must be provided by another means (for example a patched Deployment); alternatively terminate TLS at an Ingress or gateway in front of the Service. Without both variables the API is plain HTTP.Multiple operator replicas. Every replica serves the REST API and the dashboard on 8090, so a Service with several replicas can route a request to any pod. Reads are answered by any replica; a manual resolve accepted by a replica that is not the leader is completed by the leader’s Issue reconciler.
Request flow
Authentication
All/api/ requests must include the X-API-Key header with a valid key:
Roles
Roles are orderedviewer < operator < admin; each endpoint requires a minimum role (see the endpoint table). These three strings are the only valid roles: a key with any other role (for example superadmin or readonly) is accepted as a key but gets 403 on every endpoint.
viewer
Read-only. Every GET endpoint. Ideal for dashboards and observability tools.
operator
Daily ops. Everything
viewer can do, plus the POST actions (acknowledge, snooze, resolve, approve, reject, postmortem review/close/feedback) and runbook create/update.admin
Full access. Everything
operator can do, plus DELETE /runbooks/{name}.Configuring API keys
Keys are read from the operator namespace (chatcli-system by default), in this order:
- Secret
chatcli-operator-secrets, keyapi-keys(recommended) - ConfigMap
chatcli-operator-config, keyapi-keys(fallback, used only when the Secret does not exist)
{key, role, name, description} entries. name is optional: it is the identity recorded on the approval decisions the key takes (falling back to description, then to a key-<hash> fingerprint), and a quorum counts each key once, so give each approver their own key. The operator chart renders the Secret only with apiKeys.create: true (the keys are then stored in the Helm release); otherwise you create it:
Rate limiting
The API allows 30 requests per minute per client host for requests that carry no valid API key (token bucket, bursts of up to 30, refilled continuously at 0.5 req/s), and 600 requests per minute per valid API key (bursts of up to 600, refilled at 10 req/s), keyed by a digest of the key. The limit runs before authentication; the per-key limit is the same for every role. Forwarding headers are not honored, so behind a proxy every unauthenticated caller shares the proxy’s host bucket. Idle buckets are evicted after 10 minutes. When it is exceeded the operator returns429 Too Many Requests with the standard error envelope (rate limit exceeded: N requests per minute) and a Retry-After header with the seconds until the bucket holds a token again (2 for the host limit, 1 for the key limit). No X-RateLimit-* headers are sent. Implement backoff in production clients, and keep dashboards or scrapers that poll many endpoints within the budget.
CORS
Cross-origin browser requests are denied by default: no CORS headers are sent until an origin is configured. Configure it with environment variables on the operator (or the matching chart values undersecurity.*):
Allowed request headers are
Content-Type, X-API-Key and Authorization; preflight responses are cached for 1 hour. The dashboard served by the operator itself is same-origin and needs no CORS configuration.
Response format
All responses follow a Kubernetes-like pattern:- List
- Single resource
- Error
Error codes
Pagination
List endpoints support pagination via query parameters:integer
default:"1"
Page number (starts at 1)
integer
default:"20"
Items per page (values above 100 are capped at 100)
metadata.totalCount so you can compute the total number of pages. Invalid values fall back to the defaults.
Most lists accept namespace (omitted = all namespaces). Incidents, postmortems, audit events and the analytics endpoints also accept the time filters from / to (RFC 3339, for example 2026-03-19T00:00:00Z), applied to the resource’s creation time (the event time for audit). Unparseable times are ignored.
Endpoints
All paths are relative to/api/v1.
/healthz and /readyz (outside /api/v1) need no key — see Health endpoints.
Versioning
The API uses path-based versioning (/api/v1/). Only v1 exists today.
Next steps
AIOps Platform overview
How the platform detects, analyzes, and remediates incidents
Kubernetes Operator
Operator deployment, CRDs, and configuration
Incident lifecycle
Full flow: detection → analysis → remediation → resolution
AIOps in production
Cookbook: full setup with TLS, RBAC, notifications, and SLOs