> ## Documentation Index
> Fetch the complete documentation index at: https://chatcli.edilsonfreitas.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Audit and Compliance

> Immutable audit trail, granular RBAC with 4 roles, automated compliance reports, and SIEM integration for complete traceability.

The ChatCLI AIOps platform maintains an **immutable audit trail** of all actions -- from anomaly detection to remediation execution. Combined with granular RBAC and automated compliance reports, the system meets governance requirements even in regulated environments.

<Info>
  Each AuditEvent is an immutable CRD (no status). Once created, it cannot be
  modified or deleted via controllers. This ensures record integrity
  for investigations and external audits.
</Info>

## Why Audit Trail for AIOps

When a platform makes autonomous decisions on production infrastructure, traceability is no longer optional:

<CardGroup cols={2}>
  <Card title="Accountability" icon="user-shield">
    Who approved the remediation? Which AI recommended the action? When did the circuit
    breaker open? Every decision has a trail.
  </Card>

  <Card title="Post-Incident Investigation" icon="magnifying-glass">
    Reconstruct the complete timeline of an incident -- from the first signal
    to resolution -- with precise timestamps.
  </Card>

  <Card title="Regulatory Compliance" icon="scale-balanced">
    SOC2, ISO 27001, PCI-DSS: demonstrate controls over automated actions
    with immutable records and documented RBAC.
  </Card>

  <Card title="Continuous Improvement" icon="chart-line">
    MTTD, MTTR, success rate, and SLA metrics automatically calculated
    from audit events.
  </Card>
</CardGroup>

## AuditEvent CRD

The `AuditEvent` is an **immutable** CRD -- it has only `spec`, no `status`. Once created, its content is permanent.

### Complete Specification

```yaml theme={"system"}
apiVersion: platform.chatcli.io/v1alpha1
kind: AuditEvent
metadata:
  name: audit-1710856200-a7f3b2
  namespace: chatcli-system
  labels:
    platform.chatcli.io/event-type: RemediationExecuted
    platform.chatcli.io/severity: high
    platform.chatcli.io/correlation-id: inc-8f2a4b
  annotations:
    platform.chatcli.io/immutable: "true"
spec:
  # Event type
  eventType: RemediationExecuted

  # Precise timestamp
  timestamp: "2026-03-19T14:30:00.123Z"

  # Who performed the action
  actor:
    type: controller         # system | user | controller
    name: remediation-reconciler
    serviceAccount: chatcli-operator

  # Affected resource
  resource:
    apiVersion: platform.chatcli.io/v1alpha1
    kind: RemediationPlan
    name: plan-rollback-api-server
    namespace: production

  # Event-specific details
  details:
    action: Rollback
    target: deployment/api-server
    fromRevision: "6"
    toRevision: "5"
    confidence: "0.92"
    decisionMode: auto-notify
    duration: "12s"
    result: success

  # Correlation ID to group related events
  correlationID: inc-8f2a4b

  # Event severity
  severity: high
```

### Event Types (EventType)

The platform defines **20+ event types** covering the entire AIOps lifecycle.

The following events are **automatically recorded** by the operator:

* `issue_created` — when a new Issue is detected and transitions to Analyzing
* `issue_resolved` — when a remediation successfully resolves the Issue
* `issue_escalated` — when all remediation attempts fail
* `remediation_started` — when a RemediationPlan begins execution
* `remediation_completed` — when health verification confirms successful remediation
* `remediation_failed` — when a remediation fails

<Tabs>
  <Tab title="Detection">
    | EventType                 | Description                            |
    | ------------------------- | -------------------------------------- |
    | `AnomalyDetected`         | New anomaly detected by WatcherBridge  |
    | `AnomalyCorrelated`       | Anomaly correlated with existing issue |
    | `IssueCreated`            | New issue created by AnomalyReconciler |
    | `IssueEscalated`          | Issue escalated (severity elevated)    |
    | `IssueResolved`           | Issue marked as resolved               |
    | `CrossClusterCorrelation` | Cross-cluster correlation detected     |
    | `CascadeDetected`         | Staging-to-production cascade detected |
  </Tab>

  <Tab title="Analysis">
    | EventType                | Description                                    |
    | ------------------------ | ---------------------------------------------- |
    | `AIInsightRequested`     | AI analysis requested                          |
    | `AIInsightCompleted`     | AI analysis completed successfully             |
    | `AIInsightFailed`        | AI analysis failed                             |
    | `PatternMatched`         | Pattern found in Pattern Store                 |
    | `RCAEnrichmentCompleted` | Root cause enrichment completed                |
    | `ConfidenceCalculated`   | Final confidence calculated by decision engine |
  </Tab>

  <Tab title="Remediation">
    | EventType                 | Description                        |
    | ------------------------- | ---------------------------------- |
    | `RemediationPlanned`      | RemediationPlan created            |
    | `RemediationExecuted`     | Remediation action executed        |
    | `RemediationSucceeded`    | Remediation completed successfully |
    | `RemediationFailed`       | Remediation failed                 |
    | `AgenticStepExecuted`     | Agentic loop step executed         |
    | `CircuitBreakerTriggered` | Circuit breaker activated          |
  </Tab>

  <Tab title="Governance">
    | EventType                  | Description                       |
    | -------------------------- | --------------------------------- |
    | `ApprovalRequested`        | Approval requested                |
    | `ApprovalGranted`          | Approval granted                  |
    | `ApprovalRejected`         | Approval rejected                 |
    | `ApprovalExpired`          | Approval expired without response |
    | `RoleGranted`              | RBAC role granted to user         |
    | `RoleRevoked`              | RBAC role revoked from user       |
    | `ChaosExperimentStarted`   | Chaos experiment started          |
    | `ChaosExperimentCompleted` | Chaos experiment completed        |
  </Tab>
</Tabs>

### AuditActor

The `actor` field identifies who or what performed the action:

| Type         | Description                       | Example                                      |
| ------------ | --------------------------------- | -------------------------------------------- |
| `system`     | Automatic system action           | `watcher-bridge`, `correlation-engine`       |
| `controller` | Operator controller               | `remediation-reconciler`, `issue-reconciler` |
| `user`       | Human action (via kubectl or API) | `john@company.com`, `admin`                  |

### AuditResource

The `resource` field identifies the affected Kubernetes resource:

```go theme={"system"}
type AuditResource struct {
    APIVersion string `json:"apiVersion"`
    Kind       string `json:"kind"`
    Name       string `json:"name"`
    Namespace  string `json:"namespace"`
}
```

### Name Format

Each AuditEvent follows the name format:

```
audit-{unix-timestamp}-{random-6-chars}
```

Examples:

* `audit-1710856200-a7f3b2`
* `audit-1710856245-c9d4e1`
* `audit-1710856300-f2b8a6`

### Immutability Annotation

Every AuditEvent is created with the annotation `platform.chatcli.io/immutable: "true"`. An admission webhook can be configured to reject updates/deletes on resources with this annotation.

## Audit Recorder

The `AuditRecorder` is the central component that generates audit events. It offers **12 convenience functions** for the most common scenarios.

### Convenience Functions

```go theme={"system"}
type AuditRecorder struct {
    client    client.Client
    namespace string
}

// Detection
func (ar *AuditRecorder) RecordAnomalyDetected(anomaly *v1alpha1.Anomaly) error
func (ar *AuditRecorder) RecordIssueCreated(issue *v1alpha1.Issue) error
func (ar *AuditRecorder) RecordIssueResolved(issue *v1alpha1.Issue, resolution string) error

// Analysis
func (ar *AuditRecorder) RecordAIInsightCompleted(insight *v1alpha1.AIInsight) error
func (ar *AuditRecorder) RecordConfidenceCalculated(issue string, confidence float64, factors map[string]float64) error
func (ar *AuditRecorder) RecordPatternMatched(fingerprint string, issue string) error

// Remediation
func (ar *AuditRecorder) RecordRemediationExecuted(plan *v1alpha1.RemediationPlan, action string) error
func (ar *AuditRecorder) RecordRemediationResult(plan *v1alpha1.RemediationPlan, success bool) error
func (ar *AuditRecorder) RecordCircuitBreakerTriggered(namespace string, failures int) error

// Governance
func (ar *AuditRecorder) RecordApprovalDecision(request *v1alpha1.ApprovalRequest, decision string) error
func (ar *AuditRecorder) RecordRoleChange(user string, role string, action string) error
func (ar *AuditRecorder) RecordChaosExperiment(experiment *v1alpha1.ChaosExperiment, phase string) error
```

### Automatic Generation by Controllers

Controllers automatically generate AuditEvents at key points in the pipeline:

```go theme={"system"}
func (r *RemediationReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    plan := &v1alpha1.RemediationPlan{}
    r.Get(ctx, req.NamespacedName, plan)

    // Record remediation start
    r.auditRecorder.RecordRemediationExecuted(plan, plan.Spec.Actions[0].Type)

    // Execute the action
    err := r.executeAction(ctx, plan)

    if err != nil {
        // Record failure
        r.auditRecorder.RecordRemediationResult(plan, false)
        return ctrl.Result{}, err
    }

    // Record success
    r.auditRecorder.RecordRemediationResult(plan, true)
    return ctrl.Result{}, nil
}
```

### Generated Event Example

```yaml theme={"system"}
apiVersion: platform.chatcli.io/v1alpha1
kind: AuditEvent
metadata:
  name: audit-1710856200-a7f3b2
  namespace: chatcli-system
  annotations:
    platform.chatcli.io/immutable: "true"
spec:
  eventType: RemediationExecuted
  timestamp: "2026-03-19T14:30:00.123Z"
  actor:
    type: controller
    name: remediation-reconciler
    serviceAccount: chatcli-operator
  resource:
    apiVersion: platform.chatcli.io/v1alpha1
    kind: RemediationPlan
    name: plan-rollback-api-server
    namespace: production
  details:
    action: Rollback
    target: deployment/api-server
    fromRevision: "6"
    toRevision: "5"
  correlationID: inc-8f2a4b
  severity: high
```

## Compliance Reporter

The `ComplianceReporter` generates automated reports from AuditEvents, calculating essential operational metrics.

### GenerateReport

```go theme={"system"}
func (cr *ComplianceReporter) GenerateReport(
    ctx context.Context,
    namespace string,
    window time.Duration,  // E.g., 7*24h for weekly report
) (*ComplianceReport, error)
```

### Report Metrics

<Tabs>
  <Tab title="Incident Metrics">
    Incident metrics measuring detection and resolution speed.

    | Metric                          | Description                                      | Calculation                               |
    | ------------------------------- | ------------------------------------------------ | ----------------------------------------- |
    | **MTTD** (Mean Time to Detect)  | Average time between problem start and detection | `avg(anomaly.detected - anomaly.started)` |
    | **MTTR** (Mean Time to Resolve) | Average time between detection and resolution    | `avg(issue.resolved - issue.created)`     |
    | **MeanRemediationAttempts**     | Average number of remediation attempts per issue | `total_attempts / total_issues`           |

    ```yaml theme={"system"}
    incidentMetrics:
      totalIncidents: 47
      mttd: "2m15s"
      mttr: "8m30s"
      meanRemediationAttempts: 1.3
      incidentsBySeverity:
        critical: 2
        high: 8
        medium: 22
        low: 15
    ```
  </Tab>

  <Tab title="Remediation Metrics">
    Remediation metrics evaluating the effectiveness of automatic actions.

    | Metric           | Description                                               |
    | ---------------- | --------------------------------------------------------- |
    | **SuccessRate**  | Percentage of successful remediations                     |
    | **ByActionType** | Breakdown by action type (Scale, Restart, Rollback, etc.) |
    | **AutoVsManual** | Ratio of automatic vs manual remediations                 |

    ```yaml theme={"system"}
    remediationMetrics:
      totalRemediations: 52
      successRate: 0.885     # 88.5%
      byActionType:
        Rollback:
          total: 15
          successful: 14
          successRate: 0.933
        Restart:
          total: 18
          successful: 17
          successRate: 0.944
        Scale:
          total: 10
          successful: 9
          successRate: 0.900
        AdjustResources:
          total: 6
          successful: 5
          successRate: 0.833
        DeletePod:
          total: 3
          successful: 3
          successRate: 1.000
      autoVsManual:
        auto: 38
        autoWithNotification: 8
        manualApproved: 4
        manualOnly: 2
        autoPercentage: 0.885
    ```
  </Tab>

  <Tab title="SLA Metrics">
    SLA metrics verifying compliance with service level objectives.

    | Metric                   | Description                              |
    | ------------------------ | ---------------------------------------- |
    | **CompliancePercentage** | Percentage of issues resolved within SLA |
    | **Violations**           | List of SLA violations with details      |

    ```yaml theme={"system"}
    slaMetrics:
      totalIssuesInWindow: 47
      resolvedWithinSLA: 44
      compliancePercentage: 0.936    # 93.6%
      slaTargets:
        critical: 15m
        high: 30m
        medium: 2h
        low: 8h
      violations:
        - issue: issue-database-connection-pool
          severity: high
          slaTarget: 30m
          actualResolution: 45m
          reason: "Manual approval took 20 minutes"
        - issue: issue-cert-expiry-ingress
          severity: medium
          slaTarget: 2h
          actualResolution: 3h15m
          reason: "Initial remediation failed, second attempt succeeded"
        - issue: issue-disk-pressure-node-5
          severity: high
          slaTarget: 30m
          actualResolution: 1h10m
          reason: "Circuit breaker was open during the incident"
    ```
  </Tab>

  <Tab title="Approval Metrics">
    Approval system metrics tracking human governance.

    | Metric             | Description                               |
    | ------------------ | ----------------------------------------- |
    | **AutoApproved**   | Actions executed without needing approval |
    | **ManualApproved** | Actions approved by operators             |
    | **Rejected**       | Actions rejected by operators             |
    | **Expired**        | Approvals that expired without response   |

    ```yaml theme={"system"}
    approvalMetrics:
      totalApprovalRequests: 12
      autoApproved: 0          # Not applicable (auto doesn't need approval)
      manualApproved: 8
      rejected: 2
      expired: 2
      meanApprovalTime: "4m30s"
      rejectionReasons:
        - "Action too aggressive for peak hours"
        - "Prefer to investigate manually before rollback"
    ```
  </Tab>
</Tabs>

### Audit Summary

The report includes a summary of audit events generated during the period:

```yaml theme={"system"}
auditSummary:
  totalEvents: 312
  eventsByType:
    AnomalyDetected: 89
    IssueCreated: 47
    AIInsightCompleted: 45
    RemediationExecuted: 52
    RemediationSucceeded: 46
    RemediationFailed: 6
    ApprovalRequested: 12
    ApprovalGranted: 8
    ApprovalRejected: 2
    ApprovalExpired: 2
    CircuitBreakerTriggered: 1
    ChaosExperimentCompleted: 4
    PatternMatched: 23
    IssueResolved: 44
  eventsBySeverity:
    critical: 8
    high: 45
    medium: 156
    low: 103
  eventsByActor:
    system: 134
    controller: 166
    user: 12
```

## RBAC Manager

The RBAC Manager implements granular access control with **4 predefined roles**, mapped to Kubernetes ClusterRoles.

### Role Definitions

<Tabs>
  <Tab title="Viewer">
    **Viewer** -- Read-only access to all AIOps resources.

    | Resource                  | Permissions            |
    | ------------------------- | ---------------------- |
    | Anomaly, Issue, AIInsight | `get`, `list`, `watch` |
    | RemediationPlan, Runbook  | `get`, `list`, `watch` |
    | AuditEvent                | `get`, `list`, `watch` |
    | PostMortem                | `get`, `list`, `watch` |
    | ChaosExperiment           | `get`, `list`, `watch` |
    | ApprovalRequest           | `get`, `list`, `watch` |

    ```yaml theme={"system"}
    apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRole
    metadata:
      name: chatcli-aiops-viewer
      labels:
        platform.chatcli.io/rbac-role: viewer
    rules:
      - apiGroups: ["platform.chatcli.io"]
        resources: ["*"]
        verbs: ["get", "list", "watch"]
    ```
  </Tab>

  <Tab title="Operator">
    **Operator** -- Can approve remediations and manage incidents.

    | Resource        | Additional permissions (beyond Viewer)                |
    | --------------- | ----------------------------------------------------- |
    | ApprovalRequest | `update`, `patch`                                     |
    | Issue           | `update`, `patch` (change severity, manually resolve) |
    | RemediationPlan | `create` (create manual plans)                        |
    | ChaosExperiment | `create` (in allowed namespaces)                      |

    ```yaml theme={"system"}
    apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRole
    metadata:
      name: chatcli-aiops-operator
      labels:
        platform.chatcli.io/rbac-role: operator
    rules:
      - apiGroups: ["platform.chatcli.io"]
        resources: ["*"]
        verbs: ["get", "list", "watch"]
      - apiGroups: ["platform.chatcli.io"]
        resources: ["approvalrequests", "issues"]
        verbs: ["update", "patch"]
      - apiGroups: ["platform.chatcli.io"]
        resources: ["remediationplans", "chaosexperiments"]
        verbs: ["create"]
    ```
  </Tab>

  <Tab title="Admin">
    **Admin** -- Full control over AIOps, including Runbooks and configuration.

    | Resource                | Additional permissions (beyond Operator) |
    | ----------------------- | ---------------------------------------- |
    | Runbook                 | `create`, `update`, `delete`             |
    | ClusterRegistration     | `create`, `update`, `delete`             |
    | Pattern Store ConfigMap | `update`                                 |
    | ChaosExperiment         | `create`, `delete` (any namespace)       |

    ```yaml theme={"system"}
    apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRole
    metadata:
      name: chatcli-aiops-admin
      labels:
        platform.chatcli.io/rbac-role: admin
    rules:
      - apiGroups: ["platform.chatcli.io"]
        resources: ["*"]
        verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
      - apiGroups: [""]
        resources: ["configmaps"]
        resourceNames: ["chatcli-pattern-store"]
        verbs: ["update", "patch"]
    ```
  </Tab>

  <Tab title="SuperAdmin">
    **SuperAdmin** -- Unrestricted access, including RBAC management and AuditEvents.

    | Resource                   | Additional permissions (beyond Admin)           |
    | -------------------------- | ----------------------------------------------- |
    | RBAC (ClusterRoleBindings) | `create`, `update`, `delete`                    |
    | AuditEvent                 | `delete` (only for scheduled retention/cleanup) |
    | Circuit Breaker            | Manual reset via annotation                     |
    | Federation                 | Full cluster management                         |

    ```yaml theme={"system"}
    apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRole
    metadata:
      name: chatcli-aiops-superadmin
      labels:
        platform.chatcli.io/rbac-role: superadmin
    rules:
      - apiGroups: ["platform.chatcli.io"]
        resources: ["*"]
        verbs: ["*"]
      - apiGroups: ["rbac.authorization.k8s.io"]
        resources: ["clusterrolebindings"]
        verbs: ["create", "update", "delete"]
        resourceNames: ["chatcli-aiops-*"]
      - apiGroups: [""]
        resources: ["configmaps"]
        resourceNames: ["chatcli-pattern-store"]
        verbs: ["*"]
    ```
  </Tab>
</Tabs>

### EnsureRoles

The RBAC Manager ensures that ClusterRoles exist in the cluster:

```go theme={"system"}
func (rm *RBACManager) EnsureRoles(ctx context.Context) error {
    roles := []struct {
        name  string
        rules []rbacv1.PolicyRule
    }{
        {name: "chatcli-aiops-viewer", rules: viewerRules},
        {name: "chatcli-aiops-operator", rules: operatorRules},
        {name: "chatcli-aiops-admin", rules: adminRules},
        {name: "chatcli-aiops-superadmin", rules: superAdminRules},
    }

    for _, role := range roles {
        cr := &rbacv1.ClusterRole{
            ObjectMeta: metav1.ObjectMeta{Name: role.name},
            Rules:      role.rules,
        }
        _, err := controllerutil.CreateOrUpdate(ctx, rm.client, cr, func() error {
            cr.Rules = role.rules
            return nil
        })
        if err != nil {
            return fmt.Errorf("failed to create/update role %s: %w", role.name, err)
        }
    }
    return nil
}
```

### GrantRole and RevokeRole

<CodeGroup>
  ```go GrantRole theme={"system"}
  func (rm *RBACManager) GrantRole(ctx context.Context, user string, role string) error {
      binding := &rbacv1.ClusterRoleBinding{
          ObjectMeta: metav1.ObjectMeta{
              Name: fmt.Sprintf("chatcli-aiops-%s-%s", role, sanitize(user)),
          },
          RoleRef: rbacv1.RoleRef{
              APIGroup: "rbac.authorization.k8s.io",
              Kind:     "ClusterRole",
              Name:     fmt.Sprintf("chatcli-aiops-%s", role),
          },
          Subjects: []rbacv1.Subject{
              {Kind: "User", Name: user, APIGroup: "rbac.authorization.k8s.io"},
          },
      }

      _, err := controllerutil.CreateOrUpdate(ctx, rm.client, binding, func() error { return nil })
      if err != nil {
          return err
      }

      // Record in audit trail
      return rm.auditRecorder.RecordRoleChange(user, role, "granted")
  }
  ```

  ```go RevokeRole theme={"system"}
  func (rm *RBACManager) RevokeRole(ctx context.Context, user string, role string) error {
      binding := &rbacv1.ClusterRoleBinding{}
      bindingName := fmt.Sprintf("chatcli-aiops-%s-%s", role, sanitize(user))

      err := rm.client.Get(ctx, types.NamespacedName{Name: bindingName}, binding)
      if err != nil {
          return fmt.Errorf("binding not found: %w", err)
      }

      err = rm.client.Delete(ctx, binding)
      if err != nil {
          return err
      }

      // Record in audit trail
      return rm.auditRecorder.RecordRoleChange(user, role, "revoked")
  }
  ```
</CodeGroup>

## Audit REST API

The platform exposes REST endpoints for querying and exporting audit events.

### GET /api/v1/audit

Query events with filters.

**Query parameters:**

| Parameter        | Type     | Description                    | Example                |
| ---------------- | -------- | ------------------------------ | ---------------------- |
| `type`           | string   | Filter by EventType            | `RemediationExecuted`  |
| `severity`       | string   | Filter by severity             | `high`                 |
| `from`           | ISO 8601 | Start of time window           | `2026-03-18T00:00:00Z` |
| `to`             | ISO 8601 | End of time window             | `2026-03-19T23:59:59Z` |
| `actor`          | string   | Filter by actor                | `john@company.com`     |
| `correlation_id` | string   | Filter by correlation          | `inc-8f2a4b`           |
| `limit`          | int      | Maximum results (default: 100) | `50`                   |
| `offset`         | int      | Offset for pagination          | `100`                  |

**Request example:**

```bash theme={"system"}
curl -s "https://chatcli.example.com/api/v1/audit?\
type=RemediationExecuted&\
severity=high&\
from=2026-03-18T00:00:00Z&\
to=2026-03-19T23:59:59Z&\
limit=10" | jq .
```

**Response example:**

```json theme={"system"}
{
  "total": 15,
  "returned": 10,
  "events": [
    {
      "name": "audit-1710856200-a7f3b2",
      "eventType": "RemediationExecuted",
      "timestamp": "2026-03-19T14:30:00.123Z",
      "actor": {
        "type": "controller",
        "name": "remediation-reconciler"
      },
      "resource": {
        "kind": "RemediationPlan",
        "name": "plan-rollback-api-server",
        "namespace": "production"
      },
      "details": {
        "action": "Rollback",
        "target": "deployment/api-server",
        "result": "success"
      },
      "correlationID": "inc-8f2a4b",
      "severity": "high"
    }
  ]
}
```

### GET /api/v1/audit/export

Exports events in JSON format for SIEM integration. Returns a `.json` file with all events in the specified period.

```bash theme={"system"}
# Export last 24 hours
curl -s -o audit-export.json \
  "https://chatcli.example.com/api/v1/audit/export?\
from=2026-03-18T14:00:00Z&\
to=2026-03-19T14:00:00Z"

# Check export size
wc -l audit-export.json
# 312 lines (1 event per line, NDJSON format)
```

**Export format (NDJSON):**

```json theme={"system"}
{"name":"audit-1710856200-a7f3b2","eventType":"RemediationExecuted","timestamp":"2026-03-19T14:30:00.123Z","actor":{"type":"controller","name":"remediation-reconciler"},"resource":{"kind":"RemediationPlan","name":"plan-rollback-api-server","namespace":"production"},"details":{"action":"Rollback"},"correlationID":"inc-8f2a4b","severity":"high"}
{"name":"audit-1710856245-c9d4e1","eventType":"RemediationSucceeded","timestamp":"2026-03-19T14:30:12.456Z","actor":{"type":"controller","name":"remediation-reconciler"},"resource":{"kind":"RemediationPlan","name":"plan-rollback-api-server","namespace":"production"},"details":{"duration":"12s"},"correlationID":"inc-8f2a4b","severity":"high"}
```

## SIEM Integration

The platform supports event export to SIEM (Security Information and Event Management) systems such as Splunk, Elastic, and Datadog.

### Splunk

<Steps>
  <Step title="Configure HEC (HTTP Event Collector)">
    Create an HEC token in Splunk to receive events from the AIOps platform.
  </Step>

  <Step title="Create export CronJob">
    ```yaml theme={"system"}
    apiVersion: batch/v1
    kind: CronJob
    metadata:
      name: audit-export-splunk
      namespace: chatcli-system
    spec:
      schedule: "*/15 * * * *"   # Every 15 minutes
      jobTemplate:
        spec:
          template:
            spec:
              containers:
                - name: exporter
                  image: curlimages/curl:latest
                  command:
                    - /bin/sh
                    - -c
                    - |
                      # Export last 20 minutes (5 min overlap for safety)
                      FROM=$(date -u -d '20 minutes ago' +%Y-%m-%dT%H:%M:%SZ)
                      TO=$(date -u +%Y-%m-%dT%H:%M:%SZ)

                      curl -s "http://chatcli-server.chatcli-system:8080/api/v1/audit/export?from=$FROM&to=$TO" | \
                      while IFS= read -r line; do
                        curl -s -X POST \
                          "https://splunk.example.com:8088/services/collector/event" \
                          -H "Authorization: Splunk $SPLUNK_HEC_TOKEN" \
                          -d "{\"event\": $line, \"sourcetype\": \"chatcli:audit\"}"
                      done
                  env:
                    - name: SPLUNK_HEC_TOKEN
                      valueFrom:
                        secretKeyRef:
                          name: splunk-credentials
                          key: hec-token
              restartPolicy: OnFailure
    ```
  </Step>

  <Step title="Create index and dashboards">
    Configure a dedicated `chatcli_audit` index in Splunk and create dashboards
    to visualize events by type, severity, and actor.
  </Step>
</Steps>

### Elasticsearch

```yaml theme={"system"}
apiVersion: batch/v1
kind: CronJob
metadata:
  name: audit-export-elastic
  namespace: chatcli-system
spec:
  schedule: "*/15 * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          containers:
            - name: exporter
              image: curlimages/curl:latest
              command:
                - /bin/sh
                - -c
                - |
                  FROM=$(date -u -d '20 minutes ago' +%Y-%m-%dT%H:%M:%SZ)
                  TO=$(date -u +%Y-%m-%dT%H:%M:%SZ)

                  curl -s "http://chatcli-server.chatcli-system:8080/api/v1/audit/export?from=$FROM&to=$TO" | \
                  while IFS= read -r line; do
                    curl -s -X POST \
                      "https://elastic.example.com:9200/chatcli-audit/_doc" \
                      -H "Content-Type: application/json" \
                      -u "$ELASTIC_USER:$ELASTIC_PASS" \
                      -d "$line"
                  done
              env:
                - name: ELASTIC_USER
                  valueFrom:
                    secretKeyRef:
                      name: elastic-credentials
                      key: username
                - name: ELASTIC_PASS
                  valueFrom:
                    secretKeyRef:
                      name: elastic-credentials
                      key: password
          restartPolicy: OnFailure
```

## kubectl Commands

<Accordion title="Common audit queries via kubectl">
  ```bash theme={"system"}
  # List all audit events
  kubectl get auditevents -n chatcli-system

  # Filter by event type
  kubectl get auditevents -n chatcli-system \
    -l platform.chatcli.io/event-type=RemediationExecuted

  # Filter by severity
  kubectl get auditevents -n chatcli-system \
    -l platform.chatcli.io/severity=critical

  # Filter by correlation (all events for an incident)
  kubectl get auditevents -n chatcli-system \
    -l platform.chatcli.io/correlation-id=inc-8f2a4b \
    --sort-by=.spec.timestamp

  # View details of a specific event
  kubectl get auditevent audit-1710856200-a7f3b2 -n chatcli-system -o yaml

  # Count events by type (last 24h)
  kubectl get auditevents -n chatcli-system -o json | \
    jq '[.items[].spec.eventType] | group_by(.) | map({type: .[0], count: length})'

  # Check configured RBAC roles
  kubectl get clusterroles -l platform.chatcli.io/rbac-role

  # List bindings for a user
  kubectl get clusterrolebindings -l platform.chatcli.io/rbac-role \
    -o jsonpath='{range .items[*]}{.metadata.name}: {.subjects[*].name}{"\n"}{end}'

  # Generate compliance report (via API)
  curl -s "https://chatcli.example.com/api/v1/compliance/report?\
  namespace=production&window=7d" | jq .
  ```
</Accordion>

## Event Retention

<Note>
  AuditEvents are immutable but not eternal. Configure a retention policy
  to avoid excessive CR accumulation in etcd.
</Note>

```yaml theme={"system"}
# CronJob for cleaning up old events (>90 days)
apiVersion: batch/v1
kind: CronJob
metadata:
  name: audit-retention
  namespace: chatcli-system
spec:
  schedule: "0 2 * * 0"    # Every Sunday at 02:00
  jobTemplate:
    spec:
      template:
        spec:
          serviceAccountName: chatcli-superadmin
          containers:
            - name: cleanup
              image: bitnami/kubectl:latest
              command:
                - /bin/sh
                - -c
                - |
                  CUTOFF=$(date -u -d '90 days ago' +%Y-%m-%dT%H:%M:%SZ)
                  kubectl get auditevents -n chatcli-system -o json | \
                  jq -r ".items[] | select(.spec.timestamp < \"$CUTOFF\") | .metadata.name" | \
                  xargs -r kubectl delete auditevent -n chatcli-system
          restartPolicy: OnFailure
```

## Next Steps

<CardGroup cols={2}>
  <Card title="Decision Engine" icon="brain-circuit" href="/features/aiops/decision-engine">
    See how every engine decision generates AuditEvents for complete
    traceability.
  </Card>

  <Card title="Multi-Cluster Federation" icon="network-wired" href="/features/aiops/federation">
    Cross-cluster AuditEvents with unified CorrelationID.
  </Card>

  <Card title="Chaos Engineering" icon="explosion" href="/features/aiops/chaos-engineering">
    Chaos experiments generate audit events for game day compliance.
  </Card>

  <Card title="AIOps Platform" icon="brain" href="/features/aiops-platform">
    Return to the AIOps platform overview.
  </Card>
</CardGroup>