Capacity Planning and Cost Attribution

The ChatCLI AIOps platform includes three subsystems that sit next to the incident pipeline: Capacity Planner (an on-demand forecast for the resources behind your incidents), Noise Reducer (suppression of repetitive, seasonal, flapping and high-fatigue anomalies), and Cost Tracker (LLM spend per incident, served over the REST API).

Capacity Planner

The Capacity Planner produces a CPU and memory report for each Deployment that appears as the resource of an Issue. It is not a background job: it runs only when GET /api/v1/analytics/capacity is called, and it keeps no history of its own.

The planner does not read real usage and collects no usage history. It reads no metrics-server and no Prometheus data (PROMETHEUS_URL is not used here). “Current usage” is the requests of the Deployment’s first container (UsageSource: "requests"), and the percentage is requests divided by limits. Without history there is no trend to fit, so the response says Trend.Direction: "insufficient_history" and HistoryAvailable: false, and it never projects an exhaustion date. Treat the output as a requests-versus-limits check plus an incident count, not as a usage forecast.

What It Computes

For every distinct resource (kind/namespace/name) among the Issues listed (all Issues, or those in ?namespace=), the planner:

Reads the Deployment with that name and namespace. If there is none (the resource is a StatefulSet, a DaemonSet, a Pod or has been deleted), the resource is skipped with no error.
Takes the first container’s resources.limits and resources.requests for CPU and memory, and computes usage% = requests / limits × 100 (0 when there is no limit).
Correlates the resource with the incidents of its namespace inside the window (below).
Sets an urgency and one recommendation from what it knows: the requests/limits percentage and the incident correlation.

The window defaults to 7 days. When both from and to are passed (RFC 3339), the window becomes their difference, still ending now.

Trend and Urgency

UsageSource      = "requests"
HistoryAvailable = false
Trend.Direction  = "insufficient_history"   (CPUTrendPerDay = MemoryTrendPerDay = 0)
Forecast.*ExhaustionDate = null, Forecast.DaysUntil*Exhaustion = 0

Urgency = "plan" if CPU usage% >= 80 or memory usage% >= 80 or ResourceIsBottleneck
          "none" otherwise

Urgency has only these two values: without history nothing is ever urgent. The dashboard’s capacity banner lists the resources with Urgency: "plan".

Response Fields

The endpoint returns kind: CapacityForecastList with one CapacityForecast per resource. Field names are the Go names (no JSON renaming), except the nested resource reference:

CapacityForecast

Field	Type	Description
`Resource`	`{kind, name, namespace}`	The Issue’s resource
`CurrentUsage.CPUMillicores` / `.MemoryBytes`	`int64`	First container’s requests
`Limits.CPUMillicores` / `.MemoryBytes`	`int64`	First container’s limits
`UsagePercentage.CPU` / `.Memory`	`float64`	Requests divided by limits × 100
`Trend`	`ResourceTrend`	See below
`Forecast`	`ForecastResult`	See below
`IncidentCorrelation`	`IncidentResourceCorrelation`	See below
`UsageSource`	`string`	Where `CurrentUsage` comes from: always `requests`
`HistoryAvailable`	`bool`	Always `false`: no usage samples are collected
`Urgency`	`string`	`plan` or `none` (see above)

ResourceTrend

Field	Type	Description
`CPUTrendPerDay`	`float64`	Always `0` (no history to fit)
`MemoryTrendPerDay`	`float64`	Always `0` (no history to fit)
`Direction`	`string`	`insufficient_history`

ForecastResult

Field	Type	Description
`CPUExhaustionDate`	`time` or `null`	Always `null`: no exhaustion date is projected
`MemoryExhaustionDate`	`time` or `null`	Always `null`
`DaysUntilCPUExhaustion`	`int`	Always `0`
`DaysUntilMemoryExhaustion`	`int`	Always `0`
`Recommendation`	`string`	One recommendation (table below)

IncidentResourceCorrelation

Field	Type	Description
`IncidentsInWindow`	`int`	Issues created in the namespace inside the window
`ResourceRelatedIncidents`	`int`	Of those, Issues whose resource name matches
`ResourceIsBottleneck`	`bool`	`true` when `ResourceRelatedIncidents > 2`

Correlation with Incidents

For each Issue in the resource's namespace created inside the window (any state):
  IncidentsInWindow++
  If issue.spec.resource.name == resource name: ResourceRelatedIncidents++

ResourceIsBottleneck = ResourceRelatedIncidents > 2

A bottleneck raises Urgency to plan and, when requests are within limits, sets the recommendation.

Recommendation Generation

The first matching rule wins:

Condition	Recommendation
CPU or memory requests above 80% of limits	”Requests are above 80% of limits. Consider increasing limits proactively.”
`ResourceIsBottleneck`	”Repeated incidents on this resource. Review its capacity.”
Otherwise	”No usage history is collected, so no trend or exhaustion date can be projected; requests are within limits.”

How to Use

# all namespaces, last 7 days (default)
curl -H "X-API-Key: $KEY" "http://localhost:8090/api/v1/analytics/capacity"

# one namespace, 30-day window (only the difference between from and to is used)
curl -H "X-API-Key: $KEY" \
  "http://localhost:8090/api/v1/analytics/capacity?namespace=production&from=2026-08-28T00:00:00Z&to=2026-09-27T00:00:00Z"

{
  "apiVersion": "v1",
  "kind": "CapacityForecastList",
  "metadata": {"totalCount": 1, "page": 1, "pageSize": 1},
  "items": [
    {
      "Resource": {"kind": "Deployment", "name": "api-gateway", "namespace": "production"},
      "CurrentUsage": {"CPUMillicores": 900, "MemoryBytes": 536870912},
      "Limits": {"CPUMillicores": 1000, "MemoryBytes": 1073741824},
      "UsagePercentage": {"CPU": 90, "Memory": 50},
      "Trend": {"CPUTrendPerDay": 0, "MemoryTrendPerDay": 0, "Direction": "insufficient_history"},
      "Forecast": {
        "CPUExhaustionDate": null,
        "MemoryExhaustionDate": null,
        "DaysUntilCPUExhaustion": 0,
        "DaysUntilMemoryExhaustion": 0,
        "Recommendation": "Requests are above 80% of limits. Consider increasing limits proactively."
      },
      "IncidentCorrelation": {"IncidentsInWindow": 4, "ResourceRelatedIncidents": 3, "ResourceIsBottleneck": true},
      "UsageSource": "requests",
      "HistoryAvailable": false,
      "Urgency": "plan"
    }
  ]
}

The endpoint needs the viewer role (GET only). Each call lists every Issue in scope and reads one Deployment plus the namespace’s Issues per resource, so keep the calls occasional on large clusters.

Noise Reducer

The Noise Reducer runs inside the Anomaly controller. It is consulted only for an anomaly that would open a new Issue: an anomaly whose resource already has an active Issue is correlated to it first, and one that arrives within the resolution cooldown (spec.aiops.resolutionCooldownMinutes of the Instance the anomaly came from, default 10, 0 turns it off) after a resolution is suppressed before the Noise Reducer is reached. The four checks run in order and the first one that fires wins. A suppressed anomaly is marked status.correlated: true with the reason recorded as suppressed:<reason>, no Issue is created, and chatcli_operator_anomalies_processed_total{result="suppressed_noise"} is incremented. If a check fails (for example, a List error), the anomaly is not suppressed. Resources are matched by name within the anomaly’s namespace.

Strategy 1: Repetitive Suppression

Suppression condition:
  - 5 or more Anomalies in the namespace created in the last 1 hour
    with the same signalType, resource kind and resource name
    (the incoming anomaly counts)
  - and no Issue for that resource name is in Analyzing or Remediating

Result:
  - reason "Repetitive: N identical anomalies in last hour with no state change"

Severity is not part of the match.

Strategy 2: Seasonal Patterns

Every anomaly that is not suppressed is recorded as a seasonal occurrence. Patterns are stored per namespace in the ConfigMap chatcli-seasonal-patterns, key patterns, as a JSON array. SeasonalPattern (JSON):

Field	Type	Description
`signalType`	`string`	Anomaly signal type (for example `pod_restart`)
`resource`	`{kind, name, namespace}`	Resource of the first anomaly recorded for the pattern
`hourOfDay`	`int`	Hour of the day (0–23) when the pattern was first recorded
`dayOfWeek`	`string`	Weekday name (`Monday`, …)
`occurrences`	`int`	Anomalies recorded in this slot
`firstSeen` / `lastSeen`	`time`	First and last recording

Algorithm:

Recording (anomaly not suppressed):
  Find a pattern with the same signalType and resource name,
  the same weekday and |current hour - hourOfDay| <= 1
  If found: occurrences++, lastSeen = now
  Else:     append a new pattern with occurrences = 1

Suppression check:
  Same match (signalType, resource name, weekday, hour within ±1)
  AND occurrences >= 3  →  suppress
  reason "Seasonal: pattern detected on <weekday> ~<hour>:00 (<n> occurrences)"

Hours and weekdays come from the operator pod’s clock (UTC unless you set TZ on the operator; the image ships tzdata). The hour distance does not wrap around midnight (23:00 and 00:00 are 23 hours apart).

occurrences counts anomalies, not distinct weeks. Three anomalies for the same signal and resource in the same hour slot of one day are enough to suppress later ones in that slot, on that day and on the same weekday every week after. There is no confidence score and no pruning: patterns stay in the ConfigMap until you edit or delete it.

Strategy 3: Flap Detection

Flap condition:
  - 3 or more Issues for the same resource name in the namespace,
    created in the last 24 hours and currently in state Resolved

Result:
  - the new anomaly is suppressed
  - reason "Flapping: N resolved→detected cycles in 24h for this resource"

There is no separate flapping flag, hold-off timer or consolidated alert: each new anomaly is checked the same way, and suppression stops once fewer than three resolved Issues remain inside the 24-hour window.

Strategy 4: Alert Fatigue Scoring

Over the namespace's Anomalies of the same resource name in the last 24 hours:
  total      = number of anomalies
  correlated = anomalies with status.correlated = true
               (grouped into an Issue, or suppressed earlier)

  volume_score  = min(50, total * 5)
  resolve_score = int(correlated / total * 30)
  recency_score = 20, or 10 if the reference anomaly is older than 1h,
                  or 5 if older than 6h

  fatigue_score = min(100, volume_score + resolve_score + recency_score)

Suppress when fatigue_score > 80.

In practice, as few as 7 anomalies for a resource in 24 hours, all already correlated, with a recent reference anomaly, cross the threshold (35 + 30 + 20 = 85), and further anomalies for it are suppressed until the count drops.

The recency reference is the last item of the namespace’s unfiltered Anomaly list, not the newest anomaly of the resource, so recency_score can reflect another resource. The threshold > 80 is fixed and not configurable.

Cost Tracker

The Cost Tracker books the LLM spend of every incident from the usage the server reports, and serves the aggregate over the REST API.

LLM Costs per Provider

Every analysis (AnalyzeIssue, AIInsight controller) and every agentic remediation step (AgenticStep, Remediation controller) is booked from the token usage the server reports on the reply, attributed to the provider and model that actually answered (the server’s fallback chain may route elsewhere). When the reply carries no usage (a server older than the usage fields), analyses fall back to characters divided by four for both input and output, and agentic steps book zero input tokens and the reasoning length divided by four as output. The price per token is resolved in this order:

an entry for the provider in the chatcli-cost-config ConfigMap of the Issue’s namespace (the cluster operator’s word);
the shared pricing engine (llm/pricing), the same per-model tables, overrides and subscription rules the CLI’s /cost uses (including a CHATCLI_MODEL_PRICING override set on the operator process, for example through the chart’s extraEnv), so a ledger here and a session there agree on the price of the same call;
a per-provider default for a model the engine does not know (for example CLAUDEAI $3/$ 15, OPENAI $10/$ 30, OPENROUTER $2/$ 8, DEVIN $0, anything else$ 1/$3 per million input/output tokens).

Only input and output rates are applied: cache-read and cache-write discounts are not modeled on the ledger. The server also reports cost_usd / cost_known on the reply itself (see server mode); the ledger does not use those fields.

Cost Configuration

Prices are overridable per provider via the chatcli-cost-config ConfigMap, key pricing, one entry per provider name exactly as the server reports it (CLAUDEAI, OPENAI, …). The tracker looks it up in the namespace of the Issue being booked, the same namespace as its ledger, so create one in every namespace whose incidents you want to price differently:

apiVersion: v1
kind: ConfigMap
metadata:
  name: chatcli-cost-config
  namespace: production        # the namespace where the Issues live
data:
  pricing: |
    {
      "CLAUDEAI": {"InputPerMillion": 3.00, "OutputPerMillion": 15.00},
      "OPENAI":   {"InputPerMillion": 1.25, "OutputPerMillion": 10.00}
    }

An entry applies to every model of that provider; there is no per-model key. Without the ConfigMap, for a provider it does not list, or when the pricing value is not valid JSON, the pricing engine’s per-model rates apply. The ConfigMap is read on every booking, so an update applies from the next call on; calls already booked keep the price they were booked at.

IncidentCost

One ledger entry per incident, written by the AIInsight and Remediation controllers on every LLM call:

Field	Type	Description
`issueName`	`string`	Incident name
`llmCosts.totalInputTokens`	`int64`	Input tokens across all calls
`llmCosts.totalOutputTokens`	`int64`	Output tokens across all calls
`llmCosts.analysisCalls`	`int32`	`AnalyzeIssue` calls
`llmCosts.agenticSteps`	`int32`	`AgenticStep` calls
`llmCosts.estimatedCostUSD`	`float64`	Sum of every call’s cost, each priced at its own provider and model rate (pricing engine or ConfigMap override) when it was booked
`llmCosts.provider` / `llmCosts.model`	`string`	Provider and model of the latest call
`engineerTimeSavedMinutes`	`float64`	Reserved for future estimation, `0` today
`totalCostUSD`	`float64`	Equals `llmCosts.estimatedCostUSD`
`recordedAt`	`time`	When the entry was last booked

Each call is priced once, at the rates of the provider and model that answered it, and added to the entry’s total. A later call on another model (for example after a fallback) never reprices the earlier ones; provider and model only name the latest call.

Calculation example (with the CLAUDEAI override above):

Incident: CrashLoopBackOff on api-server
  - Provider: CLAUDEAI
  - LLM calls: 2 (AnalyzeIssue + 1 AgenticStep)
  - Input tokens: 8,500 | Output tokens: 2,200
  - Input cost:  8,500 / 1,000,000 * $3.00  = $0.0255
  - Output cost: 2,200 / 1,000,000 * $15.00 = $0.0330
  - estimatedCostUSD: $0.0585

The usage the server reports on each response (usage on AnalyzeIssue and AgenticStep) is what gets booked, so the ledger reflects real token counts, not estimates.

Two calls booked at the same time in one namespace do not overwrite each other: the ledger update carries the version it read, and a write conflict is retried. A booking that still fails (the ConfigMap size limit, missing RBAC, a ledger that cannot be read or an entry that cannot be decoded) is logged by the controller (Failed to book the analysis cost on the ledger or Failed to book the agentic step cost on the ledger); the analysis or step itself is not affected, and an unreadable entry is never overwritten with a fresh one.

CostSummary

Aggregation over a period, served by the REST API (viewer role, GET only), as kind: CostSummary with the summary under spec:

Field	Type	Description
`periodStart` / `periodEnd`	`time`	Period boundaries (see below)
`totalLLMCost`	`float64`	Sum of the `estimatedCostUSD` of every entry in the window
`incidentCount`	`int`	Ledger entries in the window
`costPerIncident`	`float64`	`totalLLMCost / incidentCount` (`0` when there are none)

from and to (RFC 3339) set an absolute period: with both, it is exactly [from, to]; with only from, it runs to now; with only to, it is the 30 days ending at to; with neither, it is the last 30 days. An entry is counted in full when its recordedAt (last booking) falls in the period; entries without recordedAt are always counted. Without namespace, the summary aggregates every chatcli-cost-ledger in the cluster. A namespace without a ledger returns zeros; any other failure to read a ledger returns 500 (failed to read the cost ledger: ...) instead of zeros.

# last 30 days (default), all namespaces
curl -H "X-API-Key: $KEY" "http://localhost:8090/api/v1/analytics/cost"

# one namespace, 1 to 25 September
curl -H "X-API-Key: $KEY" "http://localhost:8090/api/v1/analytics/cost?namespace=production&from=2026-09-01T00:00:00Z&to=2026-09-25T00:00:00Z"

{
  "apiVersion": "v1",
  "kind": "CostSummary",
  "spec": {
    "periodStart": "2026-09-01T00:00:00Z",
    "periodEnd": "2026-09-25T00:00:00Z",
    "totalLLMCost": 1.84,
    "incidentCount": 23,
    "costPerIncident": 0.08
  }
}

Downtime cost and return-on-investment figures are not computed: the platform has no trustworthy source for revenue per minute or engineer hourly rates, and numbers built on assumed constants would be misleading. Combine totalLLMCost with your own incident economics if you need them.

Storage Architecture (ConfigMaps)

These subsystems persist their data in ConfigMaps in the namespace of the Issue or Anomaly they concern (not the operator namespace). The Capacity Planner stores nothing.

ConfigMap	Written by	Data	Retention
`chatcli-seasonal-patterns`	Noise Reducer	Key `patterns`: JSON array of seasonal patterns	Indefinite (never pruned)
`chatcli-pattern-store`	Remediation controller (resolution learning, not the Noise Reducer)	One key per fingerprint of signal type, resource kind and severity: successful actions, success/failure counts, average resolution time, confidence boost	Indefinite (never pruned)
`chatcli-cost-ledger`	Cost Tracker	One key per incident with the `IncidentCost` JSON	Indefinite (never pruned)
`chatcli-cost-config`	You	Per-provider pricing override (`pricing` key)	User-managed

ConfigMaps have a 1MB limit in Kubernetes, roughly 3,000 ledger entries. The operator does not compact the ledger or the seasonal patterns: on a namespace with that volume, prune old keys periodically, or new bookings fail (each failure is logged, see above).

Storage Format

One ConfigMap per namespace, chatcli-cost-ledger, one key per incident holding the IncidentCost JSON. The tracker creates it with the app.kubernetes.io/managed-by: chatcli-operator label, which is how the all-namespace summary finds every ledger:

apiVersion: v1
kind: ConfigMap
metadata:
  name: chatcli-cost-ledger
  namespace: production
  labels:
    app.kubernetes.io/managed-by: chatcli-operator
data:
  api-gateway-oom-7f3a: |
    {
      "issueName": "api-gateway-oom-7f3a",
      "llmCosts": {
        "totalInputTokens": 8500,
        "totalOutputTokens": 2200,
        "analysisCalls": 1,
        "agenticSteps": 1,
        "estimatedCostUSD": 0.0585,
        "provider": "CLAUDEAI",
        "model": "claude-sonnet-5"
      },
      "engineerTimeSavedMinutes": 0,
      "totalCostUSD": 0.0585,
      "recordedAt": "2026-09-25T14:13:00Z"
    }

Entries are never compacted or expired by the operator: delete keys, or the ConfigMap, to reset a namespace’s ledger.

Integrations

REST API

/api/v1/analytics/cost serves the ledger summary; /api/v1/analytics/capacity serves the capacity forecasts; /api/v1/analytics/summary exposes incident data.

Web Dashboard

The Overview view has a capacity-warnings banner fed by /analytics/capacity: it lists the resources with Urgency: "plan" and their recommendation. The dashboard does not show the cost ledger.

Grafana

The remediation-stats.json dashboard covers remediation success by action type. No bundled dashboard covers capacity or LLM cost.

AIOps Platform

Complete AIOps pipeline architecture and how these subsystems integrate.

Get started

Documentation

Capacity Planning and Cost Attribution

Capacity Planner

What It Computes

Trend and Urgency

Response Fields

Correlation with Incidents

Recommendation Generation

How to Use

Noise Reducer

Strategy 1: Repetitive Suppression

Strategy 2: Seasonal Patterns

Strategy 3: Flap Detection

Strategy 4: Alert Fatigue Scoring

Cost Tracker

LLM Costs per Provider

Cost Configuration

IncidentCost

CostSummary

Storage Architecture (ConfigMaps)

Storage Format

Integrations

REST API

Web Dashboard

Grafana

AIOps Platform

​Capacity Planner

​What It Computes

​Trend and Urgency

​Response Fields

​Correlation with Incidents

​Recommendation Generation

​How to Use

​Noise Reducer

​Strategy 1: Repetitive Suppression

​Strategy 2: Seasonal Patterns

​Strategy 3: Flap Detection

​Strategy 4: Alert Fatigue Scoring

​Cost Tracker

​LLM Costs per Provider

​Cost Configuration

​IncidentCost

​CostSummary

​Storage Architecture (ConfigMaps)

​Storage Format

​Integrations

REST API

Web Dashboard

Grafana

AIOps Platform

Capacity Planner

What It Computes

Trend and Urgency

Response Fields

Correlation with Incidents

Recommendation Generation

How to Use

Noise Reducer

Strategy 1: Repetitive Suppression

Strategy 2: Seasonal Patterns

Strategy 3: Flap Detection

Strategy 4: Alert Fatigue Scoring

Cost Tracker

LLM Costs per Provider

Cost Configuration

IncidentCost

CostSummary

Storage Architecture (ConfigMaps)

Storage Format

Integrations