Operations
Recovery runbooks
Start with the running image digest, profile, time of failure and affected organization/endpoint or migration IDs. Preserve durable state while finding the cause. Use a bounded read request to distinguish a process failure from a dependency, permission or data-path failure.
Service unavailable or not ready
curl --silent --show-error --max-time 5 "$EDEN_ORIGIN/health"
curl --silent --show-error --max-time 5 "$EDEN_ORIGIN/ready"
kubectl -n eden get pods,pvc
kubectl -n eden get events --sort-by=.metadata.creationTimestamp| Observation | Inspect | Recovery |
|---|---|---|
| Process never opens listener | Startup error, license validation, secret formats, volume permissions and required state | Correct the input/dependency; keep the original state and keys |
| Liveness succeeds, readiness fails | Database connectivity, disk/outbox capacity, licensing, required HA membership | Restore the reported dependency; verify readiness before routing traffic |
| HA node stays unready | /health/cache-sync, /health/consensus-control, /health/nat-relay, accepted config generation and identity files | Restore trusted membership/networking; do not reset epochs or identities |
| Listener healthy, one endpoint fails | Endpoint health, backend TLS/DNS/ACL, credential tier and command restrictions | Repair that endpoint; verify a disposable request before client retry |
| Broad failures after deployment | Image/chart/config changes, schema compatibility, termination and rollout state | Follow upgrade rollback; workload rollback alone does not restore data |
Readiness failure can be deliberate durable-admission protection. Do not replace it with a liveness-only routing check or delete outbox files to make it green.
Embedded database errors
For a single-container deployment, database is locked, Failed to add Eden Node to database, or Failure while replaying WAL is a failed deployment check. A successful liveness/readiness request does not dismiss these errors.
- Verify that exactly one Eden process opens the state volume; use one replica
and Recreate replacement on Kubernetes. Check free space, write permissions and that the volume is mounted at the configured database/outbox paths.
- Keep the original volume, Turso database/WAL, DuckDB database/WAL, encryption
key and license state together. Stop the writer before taking a consistent snapshot. Do not delete WAL files, disable encryption or switch to a temporary database to make startup appear successful.
- Record the immutable image digest, non-secret paths, UTC error window and
affected organization/migration IDs. Ask Eden for the qualified replacement matching your profile; changing timeouts or repeatedly restarting is not evidence that the problem is repaired.
- After a supported replacement, verify the same organization/node/endpoint
records, a fresh disposable write/read and an idle restart. Observe the background worker logs as well as /ready. For a migration, independently verify source/target data and its durable lifecycle before cutover.
If HTTP is ready but a native Redis connection is refused, inspect both the persisted interlay and its running/listener state. Verify the same port is published by Docker or exposed through the Kubernetes Service. Container recreation must retain the state volume, configuration and license identity; do not recreate customer resources to hide missing persisted state.
Exporter failure or disk pressure
Check sink DNS/TLS/authentication, configured telemetry plane, schema and backend availability. Compare pending outbox bytes/batches, their capacities, export outcomes and the last successful export time. Check the filesystem that actually contains the outbox, DuckDB temporary directory and control state.
Restore connectivity or increase storage/capacity using your deployment's change procedure. Durable outboxes retain unacknowledged batches and can fail readiness when admission is unavailable. Their bounded in-memory producers can still drop data; a healthy durable queue is not proof of complete signal capture. After recovery, verify that backlog declines and fresh records appear in the selected backend. Replayed records can be duplicates. Do not delete, truncate or share an outbox between processes. See telemetry integrations.
An optional telemetry-query outage does not itself stop migration workers or replace their durable authority. Analysis-dependent requests with explicit require_analysis: true remain blocked until that precondition is satisfied.
Authentication or credential rotation failure
Check organization selection, token type and current membership. Login and refresh return a top-level token; refresh uses POST. An Eden token cannot replace a backend database credential or the deployment Prometheus bearer. Use authentication and database credentials to check each tier.
For OIDC, verify issuer, audience, keys, organization mapping and group/RBAC mapping. For a backend, verify target name, CA chain, hostname and required credential tier. Rotate through the endpoint/deployment's documented mechanism; restart components that load files at initialization. Test new credentials before retiring old ones. Never disable certificate verification to resolve a trust failure.
Migration stalled or failed
Read GET /api/v1/migrations/{migration} and preserve the operation ID, epoch, owner, state, heartbeat/progress timestamps, lease expiry, workers_quiesced, recoverable, terminal reason and linked workflow run. Check source authority, target availability, failed keys and live-write gap evidence.
| Durable state / evidence | Action |
|---|---|
| Deliberate operator pause without a recovery fence | Use the ordinary /resume only after repairing its cause and checking allowed lifecycle transitions |
Watchdog-stalled, recoverable: true, workers proven quiesced, supported durable checkpoint | Use the fenced /recover API with the exact operation ID; route to the recorded owner when required |
| Workers not proven quiesced | Keep replacement work blocked; restore the owning node's ability to prove termination |
operation_restart_required after a process restart | Reconcile uncertain external effects and create a fresh authorized attempt; in-place resume/restart/refresh cannot clear this fence |
| One-sided write acceptance or delivery evidence incomplete | Preserve source routing, stop cutover, inspect durable gap/quarantine evidence, reconcile both sides before a fresh attempt |
| Cutover already committed | Treat reversal as a separately validated compensating migration; do not point clients at a stale source |
For an eligible watchdog-stalled operation, the recovery body is:
{
"expected_operation_id": "00000000-0000-0000-0000-000000000000",
"mode": "resume",
"reason": "target connectivity restored",
"idempotency_key": "unique-recovery-attempt-key",
"source_quiescence_confirmed": false
}POST it to /api/v1/migrations/{migration}/recover with CONFIGURE permission. Use the real observed ID. Reuse the same key only for an identical recovery request. Checkpoint/plan identity and quiescence gates still apply; historical preview movers require the applicable source-quiescence attestation.
For a live-write gap without a restart-required fence, a quarantined PartialFailure may permit /refresh only after durable worker-quiescence proof and independently authorized target reconciliation. Refresh does not clean the target. Never clear a lease, gap marker, checkpoint, epoch or database row by hand. A source ACK can coexist with target rejection; returning source success does not prove the target is current.
See Redis support and evidence, migration operations and migration API before changing traffic or completing.
Collect a diagnostic bundle
Keep a bounded time window with UTC timestamps, product version/image digest, profile and non-secret settings, affected resource IDs, HTTP status/error code, correlation/trace IDs when present, readiness results and relevant redacted lifecycle/export events. Include source/target versions and observed capability reasons for migration failures. Record recent configuration changes and actions.
Review logs before sharing. Remove authorization headers, JWTs, licenses, passwords, connection URLs containing credentials, private keys, request bodies, Redis AUTH arguments, key/value samples and customer records. Avoid environment dumps and full Kubernetes Secret output. Share the bundle through your agreed private support channel. Retain original sensitive evidence under your own access controls if needed for the incident.