Products

Crucible Comparison Evidence

Crucible records one safe parent event for each accepted production request and one child observation for every expected contender. This enables request-level drill-down without placing high-cardinality comparison IDs in metric labels.

Parent Event

The parent records:

Comparison, run, arena, organization, and interlay identifiers.
Serving endpoint, endpoint kind, protocol family, adapter version, and dispatch-plan version.
Operation class and a process-keyed operation fingerprint.
Eden node, listener, direct route, and connection-mode context.
Expected and observed target counts.
Canonical consensus outcome and optional latency winner.
Event expiry derived from the arena retention policy.

Target Observation

Each target records:

Endpoint UUID and kind.
Comparison semantics.
Queue delay.
Dispatch latency and capture-to-completion total latency.
Success, timeout, skip, or a safe error category.
Response bytes and result counts.
Optional keyed canonical digest.
Optional LLM time-to-first-token, input/output units, and estimated cost.

Raw SQL, parameters, prompts, generated text, credentials, returned values, and raw responses are never persisted.

Reading Consensus

Two canonical targets must return the same digest.
Three or more canonical targets use majority consensus.
A tie or lack of majority is no_consensus.
A latency winner comes only from successful targets in the consensus group.
Missing target observations remain visible as incomplete.
Outcome-only and performance-only runs do not claim content consistency.

Reading Latency

Use the three timing fields together:

Field	Meaning
Queue delay	Time accepted work waited before contender dispatch.
Dispatch latency	Time spent in the contender dispatch operation; rankings use this value.
Total latency	Time from Crucible capture through observation completion.

The summary reports dispatch and total p50, p95, and p99 values, throughput, availability, errors, timeouts, expected read/write skips, queue pressure, telemetry lag, per-target read/write attempts and successes, read/write delivery gaps, and protocol-specific fields.

Protocol safety exclusions, sampling, and pause state remain visible as skips. They are not counted as failed delivery attempts. Queue loss, stale work, target failures, incomplete observations, and evidence loss remain promotion-relevant gaps.

Write Coverage

Summary and completion evidence include accepted writes, queue-dropped writes, per-target attempts and successes, known delivery gaps, incomplete observations, and evidence-sink loss. A write gap or lost telemetry event is evidence that external reconciliation is required; it is never hidden as a successful comparison.

Storage And Metrics

Detailed events are stored asynchronously in:

analytics.endpoint_comparisons
analytics.endpoint_comparison_target_observations
analytics.endpoint_comparison_delivery_gaps

Crucible is the product name. API, storage, and metric namespaces describe the underlying endpoint-comparison feature so evidence can be reused by future comparison experiences without coupling internal contracts to a product name.

Metric	Meaning
`gateway.endpoint_comparison_requests_total`	Requests accepted by a run.
`gateway.endpoint_comparison_target_latency_microseconds`	Per-contender dispatch latency.
`gateway.endpoint_comparison_target_outcomes_total`	Per-contender success, timeout, skip, and safe errors.
`gateway.endpoint_comparison_consistency_total`	Canonical consensus or noncanonical observed/partial outcomes.
`gateway.endpoint_comparison_skipped_total`	Work rejected before dispatch or by a safety rule.
`gateway.endpoint_comparison_accepted_total`	Process-wide comparison jobs accepted by the handoff queue.
`gateway.endpoint_comparison_queue_dropped_total`	Process-wide jobs rejected by the bounded handoff queue.
`gateway.endpoint_comparison_dispatch_errors_total`	Process-wide contender dispatch task failures.

Detailed retention defaults to 30 days. Summary, completion, and timeseries APIs aggregate the full retained window server-side. The request API uses a run-scoped comparison cursor to page through retained detail. While persistence is pending or has failed, Eden falls back to volatile-plus-raw evidence and surfaces the lag/drop counters. Aggregate metrics follow deployment analytics retention.

Network Effects

Current measurements are raw observed timings from the Eden node that handled production traffic. Endpoint location and connection setup can affect them.

Adapters declare whether they can support warm protocol-health and fresh TCP/TLS probes, and events already retain node, listener, route, connection mode, and adapter version. Probe scheduling and wire-adjusted rankings are future work. When introduced, adjusted latency will be a separate statistical estimate using time-matched samples and location stratification, not a replacement for raw latency or a claim of direct server execution time.