--provider ` | the key is sent to the server for that request only |
| Caller's local OAuth | `--use-local-auth [--provider CLAUDEAI\|OPENAI\|COPILOT]` | reads `~/.chatcli/auth-profiles.json`; without `--provider` tries Anthropic, OpenAI, then Copilot |
| StackSpot | `--provider STACKSPOT --client-id … --client-key … --realm … --agent-id …` | |
| Ollama | `--provider OLLAMA --ollama-url https://…` | the URL passes the [SSRF check](#ssrf-protection): HTTPS, public address. For an Ollama on a private network, set `OLLAMA_ENABLED=true` and `OLLAMA_BASE_URL` on the **server** and connect with `--provider OLLAMA` only |
The server image does not include the Devin CLI, so `DEVIN` is not available as a server provider.
## Request routing
Every prompt RPC (`SendPrompt`, `StreamPrompt`, `InteractiveSession`, `AnalyzeIssue`, `AgenticStep`) resolves its model client the same way:
* A request that forwards a credential (`--llm-key`, `--use-local-auth`, StackSpot fields, an Ollama URL) or names a provider or model gets a dedicated client for that request.
* A request that names nothing and forwards nothing goes through the [fallback chain](#fallback-chain) when one is installed, else the server's default provider and model.
* `max_tokens` is the request value when set, else the provider's `*_MAX_TOKENS` override (`ANTHROPIC_MAX_TOKENS`, `OPENAI_MAX_TOKENS`, `BEDROCK_MAX_TOKENS`, …), else the catalog ceiling of the model.
* A 401/403 from the provider on the **server's** credentials rebuilds the providers once and retries; rebuilds are throttled to one per 30 seconds process-wide. Caller-forwarded credentials are never refreshed.
* A reply stopped by a safety classifier (`stop_reason: refusal`) is resent once on the sibling model of the same provider; on a stream, only while no text has been sent yet.
Requests run concurrently, each on its own client.
### Fallback chain
```bash theme={"system"}
export ANTHROPIC_API_KEY=sk-ant-xxx OPENAI_API_KEY=sk-xxx GOOGLEAI_API_KEY=xxx
export CHATCLI_FALLBACK_MODEL_OPENAI=gpt-6-sol
chatcli server --provider CLAUDEAI --fallback-providers CLAUDEAI,OPENAI,GOOGLEAI
```
* The chain is built from `--fallback-providers` / `CHATCLI_FALLBACK_PROVIDERS` **alone**, in that order. The server does not add `--provider` to it: list the primary first yourself. The [Helm chart](/start/docker-deployment#provider-fallback) and the operator's `spec.fallback` put the primary in front for you.
* Each entry uses `CHATCLI_FALLBACK_MODEL_` when set. Without it, the primary provider (`--provider`) keeps the server model (`--model`), and any other provider runs its own default model: its model variable (for example `OPENAI_MODEL`), then the built-in default. A fallback provider never receives the primary's model id.
* A provider whose client cannot be built (no key) is skipped with a warning. The chain is installed only when **at least two** entries remain; the log then shows `Fallback chain initialized`.
* It serves only requests that bring no credentials and name no provider or model. The response names the provider and model that actually answered.
* A non-empty `CHATCLI_FALLBACK_PROVIDERS` is the only switch. `CHATCLI_FALLBACK_ENABLED` is not read by the server, and neither the Helm chart nor the operator sets it; `/config` marks it as not read when it is set.
See [Provider fallback](/providers/provider-fallback) for error classification and cooldowns.
## gRPC API
Service `chatcli.v1.ChatCLIService` (proto: `proto/chatcli/v1/chatcli.proto` in the repository), plus the standard `grpc.health.v1.Health`:
| RPC | Type | Purpose |
| - | - | - |
| `SendPrompt` | unary | One prompt, the complete reply |
| `StreamPrompt` | server stream | One prompt, the reply as the provider streams it |
| `InteractiveSession` | bidi stream | A conversation over one stream |
| `ListSessions`, `LoadSession`, `SaveSession`, `DeleteSession` | unary | Sessions stored on the server |
| `GetServerInfo` | unary | Version, provider, model, available providers, watcher, plugin/agent/skill counts, `pipeline_enabled` |
| `GetWatcherStatus` | unary | K8s watcher state |
| `Health` | unary | Liveness and version, no credential needed |
| `GetAlerts` | unary | Active watcher alerts |
| `StreamAlerts` | server stream | Watcher alerts as they are raised, with heartbeats |
| `AnalyzeIssue` | unary | Root-cause analysis and suggested actions for an Issue (operator) |
| `AgenticStep` | unary | One step of the AI remediation loop (operator) |
| `ListRemotePlugins`, `ExecuteRemotePlugin`, `DownloadPlugin` | unary / unary / server stream | Plugins installed on the server |
| `ListRemoteAgents`, `GetAgentDefinition`, `ListRemoteSkills`, `GetSkillContent` | unary | Agents and skills on the server |
| `ResolveActiveConversation`, `NewConversation`, `AppendEvent`, `ReadConversation`, `SubscribeConversation`, `SetBinding`, `ListBindings` | unary; `SubscribeConversation` server stream | [Conversation hub](/gateway/conversation-hub) |
| `ChatTurn` | unary | Full-pipeline chat turn on the server |
| `RunCoder`, `RunAgent` | server stream | Coder / agent loop on the server |
| `ListPipelineTools`, `RunPipelineTool` | unary | Tools of the server-hosted engine |
### Streaming
`StreamPrompt` forwards each fragment as the provider emits it. The final message (`done: true`) carries the provider's `usage` and `stop_reason`. When the selected route cannot stream, the reply is produced in one call and sent as a single chunk.
### Reply attribution and token usage
`SendPrompt`, `AnalyzeIssue`, `AgenticStep` and the final `StreamPrompt` message carry a `TokenUsage`, a `stop_reason`, and the `provider` and `model` that answered (a fallback chain may answer with another provider than the default):
```protobuf theme={"system"}
message TokenUsage {
int32 prompt_tokens = 1;
int32 completion_tokens = 2;
int32 cache_read_tokens = 3;
int32 cache_write_tokens = 4;
int32 reasoning_tokens = 5;
bool estimated = 6; // provider returned no usage; counts derive from length
double cost_usd = 7; // priced with the same tables /cost uses
bool cost_known = 8; // false when the model has no published price
}
```
`chatcli connect` feeds this usage to its cost tracker, so `/cost` over a remote connection prices real tokens.
### Pipeline RPCs
`SendPrompt` is a model proxy: with `chatcli connect`, memory, `/context` attachments, skills, knowledge and compaction run in **your** CLI and the server only calls the model. The pipeline RPCs make the **server** own the whole turn, with the same engine the stdio MCP and ACP servers expose:
```bash theme={"system"}
CHATCLI_SERVER_PIPELINE=true LLM_PROVIDER=CLAUDEAI chatcli server --token "$CHATCLI_SERVER_TOKEN"
```
| RPC | What runs on the server | Role |
| - | - | - |
| `ChatTurn` | one chat turn with the enrichment pipeline (`plain: true` asks for the bare passthrough); optional provider/model per turn | any authenticated caller |
| `RunCoder`, `RunAgent` | the coder / agent loop with the server's tools; the stream carries transcript lines, then the final answer | `admin` |
| `ListPipelineTools`, `RunPipelineTool` | the policy-admitted tools (`@git`, `@read`, …) | `admin` |
* The engine reads its default provider and model from `LLM_PROVIDER` / `LLM_MODEL`, not from `--provider` / `--model`.
* Sessions are namespaced by the authenticated principal (`/`), so two callers never share a conversation by picking the same id.
* Without `CHATCLI_SERVER_PIPELINE=true` the RPCs return `Unavailable: the pipeline RPCs are not enabled on this server (start it with CHATCLI_SERVER_PIPELINE=true)`.
* `GetServerInfo.pipeline_enabled` reports whether they are served.
The engine is one ChatCLI inside the server process: its workers (memory, MCP servers, scheduler) run there and its turns are serialized, so concurrent callers queue. It cannot share a process with the co-located gateway: with both `CHATCLI_SERVER_PIPELINE=true` and `CHATCLI_GATEWAY_IN_SERVER=true` the gateway runs, the pipeline stays off, and the log says `Pipeline RPCs disabled: CHATCLI_SERVER_PIPELINE and CHATCLI_GATEWAY_IN_SERVER cannot share one process; run the gateway separately`.
Helm: `pipeline.enabled: true` on the server chart. Operator: `spec.pipeline.enabled: true` on the Instance.
### AIOps RPCs
`GetAlerts`, `StreamAlerts`, `AnalyzeIssue` and `AgenticStep` feed the [operator's](/kubernetes/k8s-operator) remediation pipeline. Alerts are the [K8s watcher's](/kubernetes/k8s-watcher#alerts):
```protobuf theme={"system"}
message WatcherAlert {
string type = 1; // HighRestartCount, OOMKilled, PodNotReady, DeploymentFailing, JobFailed, CronJobMissed, NodeNotReady, ...
string severity = 2; // INFO, WARNING, CRITICAL
string message = 3;
string object = 4; // pod name, or Kind/name
string namespace = 5;
string deployment = 6;
int64 timestamp_unix = 7;
}
message GetAlertsRequest {
string namespace = 1; // optional filter
string deployment = 2; // optional filter
}
message GetAlertsResponse { repeated WatcherAlert alerts = 1; }
```
#### StreamAlerts
```protobuf theme={"system"}
rpc StreamAlerts(StreamAlertsRequest) returns (stream StreamAlertsResponse);
message StreamAlertsRequest {
string namespace = 1; // optional filter
string deployment = 2; // optional filter
bool include_current = 3; // open with the alerts active right now
}
message StreamAlertsResponse {
WatcherAlert alert = 1; // unset on a heartbeat
bool heartbeat = 2; // every 15s while the watcher is quiet
}
```
* With `include_current` the stream starts with what `GetAlerts` would return, then only new alerts follow. The watcher deduplicates by type and object within its window, so each alert is pushed once.
* Heartbeats every 15 seconds tell a quiet watcher from a dead connection. Without a watcher the stream carries heartbeats only.
* A subscriber that falls behind its bounded buffer is dropped with `ABORTED`; reopen with `include_current` to resync.
#### AnalyzeIssue
```protobuf theme={"system"}
message AnalyzeIssueRequest {
string issue_name = 1;
string namespace = 2;
string resource_kind = 3;
string resource_name = 4;
string signal_type = 5; // error_rate, oom_kill, pod_restart, ...
string severity = 6; // low, medium, high, critical
string description = 7;
int32 risk_score = 8;
string provider = 9; // optional override
string model = 10; // optional override
string kubernetes_context = 11; // deployment status, pods, events, revisions
string previous_failure_context = 12; // earlier failed remediation attempts
}
message AnalyzeIssueResponse {
string analysis = 1;
float confidence = 2; // 0.0 to 1.0
repeated string recommendations = 3;
string model = 4;
string provider = 5;
repeated SuggestedAction suggested_actions = 6;
TokenUsage usage = 7;
}
```
### Resource discovery
`ListRemotePlugins`, `ExecuteRemotePlugin` and `DownloadPlugin` expose the plugins installed on the server; `ListRemoteAgents`, `GetAgentDefinition`, `ListRemoteSkills` and `GetSkillContent` expose its agents and skills. The in-REPL `/connect` command registers the server's plugins in your session, see [Remote Connection](/server/remote-connect#remote-plugins-sessions-and-watcher-status).
## Health checks
| Check | Where | Credential | Answer |
| - | - | - | - |
| `grpc.health.v1.Health/Check` | gRPC port | none | `SERVING` once the listener serves, `NOT_SERVING` during shutdown. Service names `""` and `chatcli.v1.ChatCLIService` |
| `chatcli.v1.ChatCLIService/Health` | gRPC port | none | status and server version (what `chatcli connect` calls first) |
| `GET /healthz` | metrics port (`9090`) | none | `200 ok`; only when `--metrics-port` is not `0` |
```bash theme={"system"}
grpc-health-probe -addr=localhost:50051 # plaintext
grpc-health-probe -addr=chatcli.example.com:50051 -tls -tls-ca-cert ca.crt # TLS
curl -s http://localhost:9090/healthz
```
The health RPCs skip the bearer check, but with mTLS the TLS handshake still needs a client certificate (`grpc-health-probe -tls-client-cert … -tls-client-key …`). The container image ships `grpc-health-probe` at `/usr/local/bin/grpc-health-probe`. A kubelet `grpc` probe cannot do TLS, which is why the Helm chart probes `/healthz` and a TCP connect instead.
## Metrics
`--metrics-port` (default `9090`) serves Prometheus metrics at `/metrics` (OpenMetrics negotiation supported) and `/healthz`. The metrics listener binds **every interface** regardless of `CHATCLI_BIND_ADDRESS` and has no authentication: firewall it or set `--metrics-port 0`.
| Metric | Type | Labels |
| - | - | - |
| `chatcli_grpc_requests_total` | counter | `method`, `code` |
| `chatcli_grpc_request_duration_seconds` | histogram | `method` |
| `chatcli_grpc_in_flight_requests` | gauge | `method` |
| `chatcli_grpc_stream_messages_sent_total` / `_received_total` | counter | `method` |
| `chatcli_llm_requests_total` | counter | `provider`, `model`, `status` |
| `chatcli_llm_request_duration_seconds` | histogram | `provider`, `model` |
| `chatcli_llm_tokens_used_total` | counter | `provider`, `model`, `type` |
| `chatcli_llm_errors_total` | counter | `provider`, `model`, `error_type` |
| `chatcli_session_active_total` | gauge | — |
| `chatcli_session_operations_total` | counter | `operation` |
| `chatcli_server_info` | gauge (1) | `version`, `provider`, `model` |
| `chatcli_server_uptime_seconds` | gauge | — |
| `chatcli_watcher_*` | | see [K8s Watcher metrics](/kubernetes/k8s-watcher#prometheus-metrics-of-the-watcher) |
Go runtime and process collectors (`go_*`, `process_*`) are registered too.
### OpenTelemetry export (OTLP)
The ChatCLI engine can push its session counters to an OpenTelemetry collector over OTLP/HTTP (JSON), configured by the standard OTel variables only:
```bash theme={"system"}
export OTEL_EXPORTER_OTLP_ENDPOINT="http://otel-collector:4318"
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer%20token"
export OTEL_SERVICE_NAME="chatcli" # default
export OTEL_METRIC_EXPORT_INTERVAL=60000 # ms, default
```
In server mode this covers the turns that run on an engine inside the process, that is the [pipeline RPCs](#pipeline-rpcs) and the co-located gateway. Plain `SendPrompt`/`StreamPrompt` proxy traffic is measured by the Prometheus metrics above. Exported sums: `chatcli.llm.tokens`, `chatcli.llm.cost`, `chatcli.context.compactions`, `chatcli.context.compaction_cost`, `chatcli.cache.requests`, `chatcli.cache.storage_cost` and, only with `OTEL_RESOURCE_ATTRIBUTES` containing `chatcli.session=attr`, `chatcli.session.cost`.
## Limits and keepalive
| Setting | Default | Variable |
| - | - | - |
| Requests per second per caller | `10` | `CHATCLI_RATE_LIMIT_RPS` |
| Burst per caller | `20` | `CHATCLI_RATE_LIMIT_BURST` |
| Max message received / sent | 50 MB / 50 MB | `CHATCLI_MAX_RECV_MSG_SIZE` / `CHATCLI_MAX_SEND_MSG_SIZE` |
| Concurrent streams per connection | `100` | `CHATCLI_MAX_CONCURRENT_STREAMS` |
The rate limiter is a token bucket per caller that runs **after** authentication. It keys on the caller's subject: JWT `sub`, `mtls:`, `legacy-token` for the shared token (so every shared-token client shares **one** bucket), `system` when no credential is configured; the health RPCs are keyed by peer address. Over the limit a unary call or a stream fails with `ResourceExhausted: rate limit exceeded, retry after N seconds` and a `retry-after` header with the same N: the time one token takes to refill, rounded up and never below 1 second (`1` at the default 10 rps).
Keepalive: the server accepts client pings every 20 seconds or slower (even without active streams) and pings idle connections every 60 seconds, closing them after 10 seconds without an answer. `chatcli connect` and the operator ping every 30 seconds.
## SSRF protection
Provider URLs that a **caller** supplies (for example `--ollama-url`, fields `base_url`, `api_base`, `endpoint`, `url`, `host`, `server_url`, `realm_url`) are checked before use:
* only `https://`; `http://` requires `CHATCLI_ALLOW_HTTP_PROVIDERS=true` on the server;
* the host (or every address it resolves to) must not be in `10.0.0.0/8`, `172.16.0.0/12`, `192.168.0.0/16`, `127.0.0.0/8`, `169.254.0.0/16`, `100.64.0.0/10`, `::1`, `fc00::/7`, `fe80::/10`, `ff00::/8` or IPv4-mapped ranges; there is no override for private addresses;
* cloud metadata hostnames (`metadata.google.internal`, `metadata.goog`, `instance-data`) are refused.
The server's own configuration (for example `OLLAMA_BASE_URL`) is not subject to this check.
## Audit log
```bash theme={"system"}
export CHATCLI_AUDIT_LOG_PATH=/var/log/chatcli/audit.jsonl # must be absolute
```
Every RPC appends one JSON line to a hash-chained, file-locked trail (the same trail the LLM request auditor writes, so both kinds of entries interleave in one verifiable chain). Fields: `timestamp`, `kind` (`grpc`), `request_id`, `action` (RPC name), `actor` (`user:` or `anonymous`), `role`, `ip`, `client_id`, `method`, `resource`, `result` (`success`, `error`, `denied`), `duration`. A relative path disables the file with an error in the log. Entries also go to the structured log under the `audit` logger.
## Logs
The server uses a structured JSON logger:
* Every entry goes to the rotating file `LOG_FILE` (default `~/.chatcli/app.log`; `CHATCLI_LOG_FILE` is an alias).
* `chatcli server` and `chatcli gateway` also write the same entries as **JSON lines on stderr whenever stderr is not a terminal** — a container, a systemd unit, a pipe — so `docker logs`, `kubectl logs` and log collectors get the startup refusals and everything after them with no extra setting. On an interactive terminal the file is the only destination. `CHATCLI_LOG_STDERR=true|false` forces it either way. Stdout stays free for the stdio transports.
* `CHATCLI_ENV=dev` switches to the development console (colored, on stdout) plus the file; the stderr JSON lines are then not duplicated.
* `LOG_LEVEL` sets the level. Rotation: `LOG_MAX_SIZE` (or `CHATCLI_LOG_MAX_SIZE_MB`, in MB), `CHATCLI_LOG_MAX_BACKUPS` (default 3), `CHATCLI_LOG_MAX_AGE_DAYS` (default 28), `CHATCLI_LOG_COMPRESS` (default `true`, gzip). These are what the operator's Instance `spec.features.logRotation` sets; the server Helm chart sets them from its `logging` block (20 MB, 3 backups by default, so the log fits the pod's 200Mi data volume; see [Logs, health and metrics](/start/docker-deployment#logs-health-and-metrics)).
```bash theme={"system"}
LOG_LEVEL=debug chatcli server 2>server.log # JSON lines on stderr, since it is not a terminal
CHATCLI_ENV=dev LOG_LEVEL=debug chatcli server # colored console for development
```
In the container images the home directory is ephemeral and the distroless image has no shell to read a file with; the stderr stream is what to read there.
## gRPC reflection
Reflection needs both the `--enable-reflection` flag and `CHATCLI_GRPC_REFLECTION=true`. The flag defaults to the value of `CHATCLI_GRPC_REFLECTION`, so the env var alone turns it on (this is what the chart's `server.grpcReflection` and the Instance's `spec.server.security.enableReflection` set). Reflection calls need the same credential as any RPC:
```bash theme={"system"}
CHATCLI_GRPC_REFLECTION=true chatcli server --token "$CHATCLI_SERVER_TOKEN"
grpcurl -plaintext -H "authorization: Bearer $CHATCLI_SERVER_TOKEN" localhost:50051 list
```
Keep it off in production.
## Multiple replicas
gRPC keeps one HTTP/2 connection open, so a ClusterIP Service pins each client to one pod. With more than one replica, use a headless Service: the clients resolve `dns:///` to every pod address and balance round-robin. Helm: `service.headless: true`; the operator switches to headless automatically when `spec.replicas > 1`. Sessions, the hub database and the audit trail are per pod unless their storage is shared. With the Helm chart, several replicas on the default `ReadWriteOnce` sessions volume work only on one node; see [Rollouts on the sessions volume](/start/docker-deployment#rollouts-on-the-sessions-volume).
## Operating the server
The token is read at startup. Change it and restart:
* binary: set the new `CHATCLI_SERVER_TOKEN` and restart the process;
* Helm with `server.token`: `helm upgrade … --reset-then-reuse-values --set server.token="$(openssl rand -hex 32)"`; the Secret checksum annotation rolls the pods;
* Helm with `secrets.existingSecret`: update the Secret, then `kubectl -n chatcli rollout restart deploy/chatcli`.
Clients keep failing with `authentication failed` until they use the new token. To avoid a hard cut-over, add JWTs first (both credentials work at the same time) and retire the shared token later.
* **HS256**: one secret signs and verifies; changing `CHATCLI_JWT_SECRET` invalidates every issued token at the restart. Keep tokens short-lived.
* **RS256**: `CHATCLI_JWT_PUBLIC_KEY` may hold several PEM blocks, and a token verified by any of them is accepted. Add the new public key next to the old one, restart, switch your issuer to the new private key, and remove the old key once the old tokens have expired.
The certificate and the client CA bundle are loaded at startup. After renewing the files (or the Secret), restart the server. The operator rolls Instance pods when a referenced Secret changes; with the Helm chart run `kubectl rollout restart`.
* Binary: `/update` inside ChatCLI or your package manager, then restart the server.
* Image: pin the tag (`ghcr.io/diillson/chatcli:1.214.0`) and change it deliberately; `latest` moves.
* Helm: `helm upgrade chatcli oci://ghcr.io/diillson/charts/chatcli --version 1.214.0 -n chatcli --reset-then-reuse-values`.
`GetServerInfo` and `chatcli_server_info{version=…}` report the running version.
## Troubleshooting
| Symptom (exact text) | Cause | Fix |
| - | - | - |
| Process or container exits with status 1 and prints nothing | Stderr is a terminal (the log went to the file only), or `CHATCLI_LOG_STDERR=false` | Read `LOG_FILE`, or run with `CHATCLI_LOG_STDERR=true`; in a container the refusal is already in `docker logs`/`kubectl logs` |
| `refusing to serve an unauthenticated API on 0.0.0.0: …` | Reachable bind (Kubernetes, or `CHATCLI_BIND_ADDRESS=0.0.0.0`) with no credential | Set `CHATCLI_SERVER_TOKEN`, JWT material or a client CA; or bind `127.0.0.1` |
| `refusing to start: CHATCLI_JWT_PUBLIC_KEY is set but no RSA public key could be loaded: …` | Wrong path or not a PEM public key, and JWT is the only credential | Fix the key (PEM `PUBLIC KEY`, `RSA PUBLIC KEY` or `CERTIFICATE`) |
| `FATAL: TLS certificate load failed: … (cert=…, key=…)` | Cert or key missing, unreadable or mismatched | Check the paths and that the key matches the certificate |
| `FATAL: --tls-client-ca requires --tls-cert and --tls-key` | mTLS without server TLS | Add the server certificate and key |
| `FATAL: --tls-cert is set (…) but --tls-key is not: refusing to start a plaintext listener` (or `--tls-key is set … but --tls-cert is not`) | Half a TLS pair | Set both paths, or neither for plaintext |
| Client: `transport: authentication handshake failed: tls: first record does not look like a TLS handshake` | Client dials TLS, server is plaintext | `CHATCLI_ALLOW_INSECURE=true`, or enable TLS on the server |
| Client: `error reading server preface: remote error: tls: certificate required` | Server enforces mTLS and the client sent no certificate | Set `CHATCLI_TLS_CLIENT_CERT` and `CHATCLI_TLS_CLIENT_KEY` and use `--tls` |
| Client: `x509: certificate signed by unknown authority` | Server certificate not trusted | `--ca-cert ca.crt` |
| Client: `x509: certificate is valid for …, not …` | The dialed name is not in the SANs | Dial a name in the certificate, or reissue it with that SAN |
| Client: `connect: connection refused` | Server not running, wrong port, or bound to loopback of another host/container | Check `CHATCLI_BIND_ADDRESS` and the port |
| `Connected to ChatCLI server (version: …, provider: remote, model: remote)`, then `Unauthenticated desc = authentication failed` | Wrong or missing token (the health check does not need one, `GetServerInfo` does) | Pass the right `--token` / `CHATCLI_REMOTE_TOKEN` |
| `Unauthenticated desc = authentication failed` with a token that works otherwise; the server log shows `auth failure rate limit exceeded` | Something on the same client host failed authentication more than 5 times in a minute (a wrong token in another script, an expired JWT) and exhausted the host's failure budget | Fix the failing caller; the budget refills at one slot per 12 s and the table clears every 5 minutes |
| `Unauthenticated desc = token expired` | JWT `exp` in the past (beyond 30 s skew) | Issue a new token; check clocks |
| `Unauthenticated desc = JWT verification material failed to load; the server accepts no credentials until it is fixed` | JWT key failed to load | Fix the key and restart |
| `ResourceExhausted desc = rate limit exceeded, retry after 1 seconds` | Caller over `CHATCLI_RATE_LIMIT_RPS`/`BURST`; shared-token callers share one bucket | Raise the limits or give callers their own JWT `sub` |
| `PermissionDenied desc = coder, agent and tool runs execute on the server and require the admin role` | JWT/mTLS caller without `admin` on an exec pipeline RPC | Use an admin credential |
| `Unavailable desc = the pipeline RPCs are not enabled on this server (start it with CHATCLI_SERVER_PIPELINE=true)` | Pipeline off | Set `CHATCLI_SERVER_PIPELINE=true` (and not `CHATCLI_GATEWAY_IN_SERVER`) |
| `invalid provider configuration: provider_config[base_url]: non-HTTPS provider URLs are blocked; set CHATCLI_ALLOW_HTTP_PROVIDERS=true to allow` | Caller sent an `http://` URL (for example `--ollama-url`) | Use HTTPS, set the variable on the server, or configure the URL on the server |
| `… resolves to blocked IP …` | Caller-supplied URL points to a private address | Configure that endpoint on the server (`OLLAMA_BASE_URL`) instead |
| `Fallback chain initialized` never appears | Fewer than two providers in the list have working credentials | Add keys, or list more providers |
| Container `unhealthy` with TLS on | The image `HEALTHCHECK` probes plaintext | Override it with `grpc-health-probe -tls …` |
## Next steps
Connect to the server
Run the server in a container or with Helm
Kubernetes context for every prompt
Managed Instances and AIOps
This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.