Operations

Monitoring and alerting

Configure an exporter, then verify names, units, labels, retention and freshness in your actual backend before building alerts. The signal reference and complete catalogs describe optional producers as well as Eden Service. A missing optional series does not mean its value is zero.

Operator dashboard

Panel	Evidence	Interpretation
Availability	`/health`, `/ready`, scrape success and external request probe	Separate a live process from a service admitting safe work
Traffic and errors	Gateway operation/command counts and errors, HTTP status, application success	`gateway_requests_total` counts received batches, not individual Redis operations
Latency	Gateway operation/Redis command latency and backend latency	Distributions here use microseconds; compare equivalent workload and batching
Export delivery	Outbox bytes/batches and capacities, outcomes, last successful export	Queue growth and stale exports precede durable-admission pressure
Redis saturation	Managed lease active/rejection/cancellation series, endpoint connection limits	Separate admission pressure from backend faults and client cancellations
Migration	Durable operation progress/heartbeat, failed batches, mismatches, replay gaps, cutover preflight	Running state or source scanning alone does not prove sink progress
HA	Cache-sync, consensus-control, NAT relay, participant digest and lease health	Process liveness alone does not prove stage authority or membership
Host/storage	CPU quota/use, RSS, filesystem free bytes/inodes and disk latency	Collect with your platform/node agent; do not assume Eden supplies every host metric

Start with per-instance and per-endpoint views. Aggregate only when label sets, units and signal meanings match. Keep customer keys, SQL text, tokens and unbounded command arguments out of metric labels.

Example Prometheus rules

The following rules use literal native export names without dotted prefixes. Adjust job to your scrape configuration. OTLP Collector or backend translation can rename metrics; match the observed names before installing the rules. The 80%/5-minute values are example operating thresholds, not product limits.

yaml

groups:
  - name: eden-operations
    rules:
      - alert: EdenScrapeFailed
        expr: up{job="eden"} == 0
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: Eden metrics scrape is failing
      - alert: EdenScrapeTargetMissing
        expr: absent(up{job="eden"})
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: No Eden scrape target is present
      - alert: EdenTelemetryOutboxHigh
        expr: >-
          (eden_telemetry_outbox_bytes{job="eden"}
          / eden_telemetry_outbox_capacity_bytes{job="eden"} > 0.8)
          and (eden_telemetry_outbox_capacity_bytes{job="eden"} > 0)
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: Eden telemetry durable outbox exceeds 80 percent
      - alert: EdenRedisLeaseRejected
        expr: increase(gateway_redis_lease_rejections_total{job="eden"}[5m]) > 0
        for: 2m
        labels:
          severity: warning
        annotations:
          summary: Redis managed lease admission is rejecting work
      - alert: EdenMigrationBatchFailed
        expr: increase(migration_batches_failed_total{job="eden"}[5m]) > 0
        labels:
          severity: critical
        annotations:
          summary: A migration batch permanently failed

Validate with promtool check rules and test notifications before relying on them. Prometheus alerting rules describe evaluation and the for interval. Scrape failure also covers bad Prometheus credentials and the configured response-size bound; pair it with external HTTP/readiness probes so the monitoring path is independently checked.

Add export-staleness and outbox batch-capacity alerts as well as byte capacity. Only evaluate an export timestamp while work is pending; before the first export it can be zero, and idle exporters need not advance. Align labels explicitly when combining signals. For audit and metering, use their separate delivery status/outbox surfaces; telemetry delivery does not establish audit delivery.

Latency and SLO queries

Native direct Prometheus distributions expose count/sum summaries without quantile buckets. For a traffic-bearing interval, this query computes mean Redis command latency in milliseconds:

promql

sum(rate(gateway_redis_command_duration_microseconds_sum{job="eden"}[5m]))
/
sum(rate(gateway_redis_command_duration_microseconds_count{job="eden"}[5m]))
/ 1000

Guard zero-traffic windows in dashboards. Do not use histogram_quantile on this summary or call a mean p99. OTLP exports exponential histograms; percentile queries depend on your collector/backend's supported histogram conversion. Keep that path's units and count semantics intact and validate it with real traffic.

Define availability from successful customer operations, with explicit exclusions and an error budget. Set paging thresholds using observed normal traffic, business requirements and sustained violations. Mismatches and live-write gaps are correctness evidence: inspect durable migration state immediately and keep cutover blocked rather than averaging them into a general error-rate SLO.

Trace and log investigation

Find eden.http.request and handler/backend spans by time and resource context, then correlate available trace/request IDs with structured logs and durable migration operation/run IDs. Retained telemetry queries require an organization JWT; the Prometheus bearer grants deployment-level scrape access instead.

Keep log collection active through your container/platform agent when using OTLP: Eden's native OTLP path exports metrics and traces, not OTLP log records. Use a bounded diagnostic log level and your retention/access policy. Follow Recovery runbooks for export, authentication, readiness or migration incidents, and verify signal freshness again after remediation.