Connect Eden to telemetry services

Eden Service can keep local telemetry and export operational copies to customer collectors. Choose destinations per signal; selecting an analytics query provider does not enable an exporter. For instrument names and record schemas, see signal reference and metrics.

Supported destinations

Jaeger, Tempo, Honeycomb, New Relic, Grafana Cloud, and other backends can be reached through a compatible customer-managed OTLP collector. They are not additional native Eden exporters. The Datadog endpoint connector under endpoints/datadog is an API integration, separate from runtime telemetry.

Configuration and defaults

Use [telemetry] in eden.toml. Configuration precedence is compiled defaults, TOML, legacy flat environment variables, nested EDEN__… variables, then CLI overrides. The examples below show the existing flat aliases where available. Set the values and restart the process to install or change exporter tasks. The telemetry-export Cargo feature is required for ClickHouse/DuckDB, OTLP, DogStatsD, and native span fanout. Prometheus is separately controlled by its runtime switch; the optional ShardTelemetry adapter also needs its build overlay.

Defaults: durable store on, logs on at info, backtraces off, OTLP, DogStatsD and Prometheus off. ClickHouse is the default query provider. Deployment profiles may override these values. Keep local durability enabled when adding external operational copies unless your profile explicitly supports a different retention design.

Prometheus and Grafana

Mount a protected file containing a random token with at least 32 non-whitespace bytes into Eden and into Prometheus. Eden loads it at startup. Set:

EDEN_PROMETHEUS_ENABLED=true
EDEN_PROMETHEUS_BEARER_TOKEN_FILE=/run/secrets/prometheus/token
EDEN_PROMETHEUS_MAX_RESPONSE_BYTES=16777216

Prometheus configuration (paths are inside the Prometheus container), using the supported file-based authorization setting:

scrape_configs:
  - job_name: eden-service
    scrape_interval: 15s
    metrics_path: /metrics
    authorization:
      type: Bearer
      credentials_file: /run/secrets/prometheus/token
    static_configs:
      - targets: [eden-service:8000]

Use a private service and TLS when crossing a trust boundary. /metrics is process-wide, including tenant identifiers; it is not the organization-filtered analytics API. Disabled returns 404, missing/incorrect bearer returns 401, and an oversized response returns 503 without a partial body. The default 16 MiB response bound is configurable from 64 KiB to 64 MiB.

Add Prometheus as the Grafana data source. Inspect the live # HELP and # TYPE lines for exact names and types before writing queries. Counters are cumulative; use rate() with a suitable range, and account for process restarts. Avoid adding a second eden prefix to Eden's already-prefixed names. Some dashboards use the separate Collector scrape endpoint in grafana/otel-collector-config.yaml; that pipeline has its own namespace and temporality conversion and is not the same naming contract as Eden's direct scrape.

The current direct text writer preserves dots in metric prefixes. If your receiver requires legacy Prometheus identifiers, use an OTLP-to-Prometheus Collector pipeline with name translation and verify the translated names. The signal naming reference distinguishes literal names from historical logical names.

OpenTelemetry / OTLP

Expose both OTLP protocols on your collector, normally gRPC 4317 and HTTP 4318. Eden uses origins without a signal path and appends /v1/metrics and /v1/traces itself. Do not configure those suffixes in Eden.

EDEN_OTLP_EXPORT_ENABLED=true
EDEN_OTLP_COLLECTOR=http://otel-collector:4317
EDEN_OTLP_METRICS_ENDPOINT=http://otel-collector:4318
EDEN_OTLP_TRACES_ENDPOINT=http://otel-collector:4318
EDEN_OTLP_EXPORT_INTERVAL_MS=2000
EDEN_OTLP_EXPORT_TIMEOUT_MS=10000

If HTTP origins are omitted, Eden derives them from the collector origin, changing port 4317 to 4318. Origins must use HTTP(S) and have no credentials, path, query, or fragment. Export interval range: 100–300,000 ms; per-request timeout: 100–60,000 ms. The resource service name is eden, native scope name eden-service, and exports carry node_uuid.

For authentication, mount a private JSON string map, for example:

{"authorization":"Bearer REPLACE_WITH_COLLECTOR_TOKEN"}

Set EDEN_OTLP_HEADERS_FILE=/run/secrets/otel/headers.json. The file is loaded once; restart Eden after rotating it. Maximum 64 KiB, 32 headers, 8 KiB per value. content-type, content-encoding, content-length, and host are exporter-managed and rejected. Header values are not logged. The same configured headers must work for both HTTP exports and gRPC metadata.

Minimal collector configuration for checking arrival (the debug exporter is for an isolated setup; replace it with your backend export before production):

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318
processors:
  batch: {}
exporters:
  debug:
    verbosity: basic
service:
  pipelines:
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [debug]
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [debug]

Logs need a separate collector/container integration. A logs pipeline with only an OTLP receiver will not make Eden start sending logs over OTLP. Both native spans and tracing instrumentation can appear in your trace backend; they come from distinct producers, not duplicate retention guarantees.

Datadog

Run a Datadog Agent reachable from Eden. Enable the Agent's DogStatsD listener, OTLP gRPC/HTTP receivers and container-log collection as needed. Datadog documents the Agent's OTLP receivers and ports. Set the native Eden switches:

EDEN_DOGSTATSD_ENABLED=true
EDEN_DOGSTATSD_ENDPOINT=datadog-agent:8125
EDEN_OTLP_EXPORT_ENABLED=true
EDEN_OTLP_COLLECTOR=http://datadog-agent:4317
EDEN_OTLP_METRICS_ENDPOINT=http://datadog-agent:4318
EDEN_OTLP_TRACES_ENDPOINT=http://datadog-agent:4318

DogStatsD exports metric deltas every 10 seconds in UDP packets up to 1,432 bytes, with service:eden and node_uuid tags. It has no acknowledgement or durable retry guarantee. Native distributions export delta count/sum where supported; do not assume they become Datadog distribution samples or preserve all source quantiles. OTLP carries the richer metric and span representations. If you enable both metric paths, choose which series each dashboard uses so it does not count the same workload twice.

Mount Datadog credentials only into the Agent. Eden does not need DD_API_KEY. Collect both container output streams: structured records reach Eden's durable sink in process, while console formatting defaults to the logger's display format and tracing diagnostics use their subscriber's formatter. Do not assume every console line is JSON merely because the retained log row is structured.

ClickHouse and DuckDB

External datastore profile:

[databases.clickhouse]
url = "http://clickhouse:8123"
username = "eden_telemetry"
password = "REPLACE_USING_DEPLOYMENT_SECRET"

[telemetry]
clickhouse_enabled = true
clickhouse_plane = "private"
clickhouse_schema_management_enabled = true
clickhouse_collection_interval_ms = 2000
clickhouse_live_collection_interval_ms = 1000
clickhouse_batch_size = 512
clickhouse_log_channel_capacity = 8192

[telemetry.clickhouse_outbox]
path = "/var/lib/eden/clickhouse-telemetry.redb"
max_events = 100000
max_bytes = 536870912
export_batch_size = 128
export_interval_ms = 500

Provision credentials through your deployment's secret mechanism, and use an HTTPS ClickHouse origin where appropriate. Schema management needs DDL rights; to provision schemas separately, set clickhouse_schema_management_enabled = false and bootstrap the matching schema version with eden-clickhouse-bootstrap. Private telemetry uses analytics; public_marketplace uses the physically separate analytics_public_marketplace database and plane-scoped credentials.

Persist the outbox directory on a volume owned by the Eden process. The outbox contains batches, even though its capacity field is named max_events. Acknowledged batches are removed; pending batches replay after restart with at-least-once semantics, so duplicates are possible. Bounded in-memory admission can reject records before persistence. Older queued data is not evicted to admit new data when the outbox is full. Alert on export loss, queue size, and last successful acknowledgement using the outbox metrics.

In embedded builds the same clickhouse_enabled switch and exporter are used for DuckDB retention, without an external ClickHouse server. Use [telemetry.duckdb] for file, memory, temporary storage, retention and row limits; see deployment profiles. This setting does not create an external ClickHouse connection in that profile.

CloudWatch

The CloudWatch backend in Service reads metrics and structured log/trace records. It does not install a periodic PutMetricData/PutLogEvents exporter. Use a customer-managed ADOT/OpenTelemetry Collector or log agent to ingest the signals into AWS, preserving the organization and node dimensions and structured log fields expected by Eden.

AWS documents CloudWatch Agent OTLP ingestion. That enables AWS-side ingestion; verify separately that metric dimensions and log/trace record shapes match Eden's query adapter. Sending traces to X-Ray alone does not populate the trace log group queried by this adapter.

[telemetry]
provider = "cloudwatch"

[telemetry.cloudwatch]
enabled = true
region = "us-east-1"
metric_namespace = "Eden"
metric_names = "request_duration_ms,cpu_time_ms,memory_usage_bytes,active_connections,requests,errors"
log_group_name = "/eden/service"
trace_log_group_name = "/eden/traces"
query_period_seconds = 60

The listed metric names are query defaults, not additional native Eden instruments; configure them to match what your collector actually writes. The Service build needs aws. Its AWS identity needs permissions to read CloudWatch metrics (cloudwatch:GetMetricData) and the selected log groups (logs:FilterLogEvents). Use the deployment's normal AWS credential chain. The ingestion agent needs its own write permissions. These are separate from access granted to users of Eden's organization-scoped telemetry APIs.

ShardTelemetry

This provider requires the private adapter build overlay in addition to runtime configuration. Setting a flag on a public binary cannot install it.

[telemetry]
provider = "shard_telemetry"

[telemetry.shard_telemetry]
enabled = true
mode = "embedded"
data_directory = "/var/lib/eden/shard-telemetry"
telemetry_plane = "private"

Embedded mode supports locally retained metrics, logs and Eden HTTP traces. remote and embedded_and_forward support metric/log delivery but not Eden's HTTP trace producer. They additionally require a host:port remote endpoint and private auth_token_file. Configure capacity and retention budgets using the provider contract. Inspect provider status for health, freshness, completeness, drops and queryable watermarks: installation or acknowledgement alone does not prove complete query-visible evidence.

Request audit exports: ClickHouse and S3

Request audit events use [analytics.audit] and a separate durable outbox. They are not all operational logs. Multiple existing sinks can be configured:

[analytics.audit]
enabled = true

[[analytics.audit.sinks]]
kind = "clickhouse"
table = "analytics.audit_trail"

[[analytics.audit.sinks]]
kind = "clickhouse"
url = "https://audit-clickhouse.example.com:8443"
database = "audit"
user = "eden_audit"
password = "REPLACE_USING_DEPLOYMENT_SECRET"
table = "audit.audit_trail"

[[analytics.audit.sinks]]
kind = "s3"
provider = "aws_s3"
region = "us-east-1"
bucket = "example-eden-audit"
prefix = "eden/"

S3-compatible destinations use provider = "generic_s3", endpoint_url and, where needed, force_path_style; prefer runtime credentials to plaintext static keys. External ClickHouse audit tables must match AuditEventRow; this sink does not provision an arbitrary external schema. At most one internal ClickHouse sink can be configured. Internal sinks accept table only; set url to use separate connection settings. If asynchronous ClickHouse inserts are configured, require wait_for_async_insert=1 with async_insert=1.

Each sink acknowledges independently. See request audit configuration for sampling, outbox sizing, retries, object layout, and readiness behavior. Use event_id when reconciling at-least-once audit delivery. Audit fanout is distinct from the correctness-critical migration write log.

Helm and standalone runner

The Eden chart exposes these values (collectors are customer-managed):

opentelemetry:
  enabled: true
  grpcEndpoint: http://otel-collector:4317
  httpEndpoint: http://otel-collector:4318
  headersSecret:
    secretName: eden-otel-headers
    secretKey: headers.json
prometheus:
  enabled: true
  bearerTokenSecret:
    secretName: eden-prometheus
    secretKey: token
datadog:
  enabled: false

The Datadog profile enables DogStatsD/OTLP plus log autodiscovery. When generic OpenTelemetry and Datadog are both enabled, generic OTLP origins take precedence while DogStatsD and log discovery still target Datadog. See the chart's observability exporter contract.

For the customer runner, set exporter switches in EDEN_SERVICE_ENV_FILE; configure EDEN_SERVICE_OTLP_HEADERS_HOST_FILE and EDEN_SERVICE_PROMETHEUS_TOKEN_HOST_FILE for secret mounts. Files must be owner-readable regular files and are mounted read-only, without copying values into container environment variables.

Verify delivery

1. Fetch the live API document and confirm your image's features.
2. Generate authorized endpoint traffic; idle dynamic metric families can have

no series, and traces may be sampled or buffered.

1. Check the direct Prometheus target or the destination collector for the

expected service, node, signal and metric names.

1. Query retained metrics, logs and traces through the

tenant-scoped API. External collector success does not select the dashboard's query backend.

1. Check loss/queue metrics and provider completeness before using telemetry

as migration or compliance evidence.

The first exporter task group shares a 20-second shutdown deadline, followed by a five-second OpenTelemetry SDK shutdown. Later ClickHouse metric, log and outbox phases have additional bounded waits; 20 seconds is not a bound on the whole process shutdown. External operational exports are not a replacement for a durable store, and shutdown cannot guarantee delivery after capacity rejection or an unreachable destination.

Sources: startup wiring, configuration and defaults, environment aliases, exporters, audit configuration, CloudWatch queries.