OpenTelemetry (OTLP)

BetterDB Monitor speaks OTLP in both directions. It ingests traces from your instrumented applications and metrics pushed by an OpenTelemetry Collector for instances it doesn’t connect to, and it can export its metrics and events to any OTLP collector.

One thing to be clear about up front: Monitor’s metrics are Prometheus-first (see Prometheus Integration). The OTLP metrics export is a mirror of that registry, not the source of truth. And on the trace side Monitor is a receiver: it stores spans your apps send, it does not emit spans of its own.

At a glance

Signal Direction Endpoint Default
Traces Ingest (receive) POST /v1/traces on
Metrics (external instances) Ingest (receive) POST /v1/external/metrics on
Metrics Export (mirror) ${OTLP endpoint}/v1/metrics off, opt-in
Events Export (logs) ${OTLP endpoint}/v1/logs off, opt-in

Both exports are off until you set OTEL_EXPORTER_OTLP_ENDPOINT. The collector handles fan-out to Jaeger, Tempo, Cloudwatch, or whatever backend you run.

Trace ingestion

Monitor exposes a standard OTLP/HTTP trace receiver at /v1/traces. Note this path sits at the server root, not under the /api prefix.

It accepts both OTLP encodings:

  • application/x-protobuf - the OTel SDK default (OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf)
  • application/json - set OTEL_EXPORTER_OTLP_PROTOCOL=http/json

Ingested spans from @betterdb/* instrumentation scopes (plus their root spans) are kept and rendered as request waterfalls in the AI Traces view, correlated with the live Valkey state underneath each request.

Point your application’s OTLP exporter at Monitor:

OTEL_EXPORTER_OTLP_ENDPOINT=http://monitor-host:3001
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf          # or http/json
# only if you set OTEL_INGEST_TOKEN on Monitor:
OTEL_EXPORTER_OTLP_HEADERS=Authorization=Bearer <token>

Auth. If OTEL_INGEST_TOKEN is set, requests must carry Authorization: Bearer <token>. The token is required in cloud mode, where /v1/traces is allowlisted past session auth. Set OTEL_INGEST_ENABLED to false, 0, no or off to turn the receiver off entirely.

The same OTEL_INGEST_ENABLED gate and OTEL_INGEST_TOKEN also cover POST /v1/external/metrics — see Metrics ingestion below.

Metrics ingestion

BetterDB Monitor can also ingest OTLP metrics, at POST /v1/external/metrics, for instances it doesn’t dial itself — a Redis or Valkey behind a firewall, or one whose credentials you’d rather not hand to Monitor. An OpenTelemetry Collector scrapes the instance and pushes metrics here instead.

Register the instance first. Add Connection → OTLP push, giving the host and port the collector reports. Metrics for an instance that isn’t registered are dropped (unknown_instance); metrics for an instance BetterDB already polls directly are also dropped (already_polled). If an unregistered instance pushes anyway, BetterDB lists it under “N discovered via OTLP” in the connection switcher (or below it when you have one connection or none). Register opens the OTLP push dialog prefilled with the name (service.name), host and port; Dismiss hides it for 24 hours. The list lives in memory: it holds at most 200 instances, forgets one after 24 hours without a push, and resets on restart. Set OTLP_DISCOVER_INSTANCES to false, 0, no or off to turn discovery off. Points from unregistered instances are still dropped until you register them. When workspace auth is on, only admins see the list.

Untrusted networks: the discovered list reflects whatever reaches the ingest endpoint. If peers you don’t trust can reach it, set OTEL_INGEST_TOKEN so only authenticated pushes are recorded.

Identity. Each ResourceMetrics payload is matched to a connection by service.instance.id (host:port, or Valkey Admin’s host-port form), falling back to server.address plus server.port. redisreceiver doesn’t set server.address/server.port by default, so the collector config needs resource_attributes.server.address.enabled: true and server.port.enabled: true.

Collector config. The Add Connection → OTLP push tab generates this config with your host, port and this server’s URL filled in:

receivers:
  redis:
    endpoint: <host>:<port>
    collection_interval: 15s
    resource_attributes:
      server.address:
        enabled: true
      server.port:
        enabled: true
    metrics:
      redis.maxmemory:
        enabled: true
      redis.role:
        enabled: true
      redis.cmd.calls:
        enabled: true
      redis.cmd.usec:
        enabled: true

exporters:
  otlphttp/betterdb:
    metrics_endpoint: <betterdb-url>/v1/external/metrics
    headers:
      Authorization: 'Bearer ${env:BETTERDB_OTEL_INGEST_TOKEN}'

service:
  pipelines:
    metrics:
      receivers: [redis]
      exporters: [otlphttp/betterdb]

Use metrics_endpoint, not the plain endpoint — endpoint appends the standard /v1/metrics path and misses this route entirely.

Compression. Request bodies compressed with gzip (the otlphttp exporter’s default) or deflate are accepted on both /v1/external/metrics and /v1/traces; any other Content-Encoding is rejected with 415. Request bodies are limited to 1 MiB after decompression, and oversize requests are rejected; if the collector config adds a batch processor, keep send_batch_max_size small enough to stay under this limit.

Supported vocabularies. Monitor understands the redis.* names emitted by the collector-contrib redisreceiver, and the equivalent valkey.* names from Valkey Admin’s exporter (only valkey.memory.used and valkey.cpu.time are confirmed against a real deployment so far; the rest of the valkey.* mapping assumes it mirrors redisreceiver). Only cumulative sums and gauges are accepted — histograms, summaries and delta-temporality sums are rejected outright (unsupported_type / unsupported_temporality). Each accepted point maps to exactly one INFO field; nothing is summed:

OTLP metric (redis. or valkey.) Point attributes INFO section.field
memory.used — memory.used_memory
memory.rss — memory.used_memory_rss
memory.peak — memory.used_memory_peak
memory.lua — memory.used_memory_lua
memory.fragmentation_ratio — memory.mem_fragmentation_ratio
memory.used_memory_overhead — memory.used_memory_overhead
memory.used_memory_startup — memory.used_memory_startup
maxmemory — memory.maxmemory
cpu.time state ∈ sys, sys_children, sys_main_thread, user, user_children, user_main_thread cpu.used_cpu_<state>
commands — stats.instantaneous_ops_per_sec
commands.processed — stats.total_commands_processed
connections.received — stats.total_connections_received
connections.rejected — stats.rejected_connections
keys.evicted — stats.evicted_keys
keys.expired — stats.expired_keys
keyspace.hits — stats.keyspace_hits
keyspace.misses — stats.keyspace_misses
net.input — stats.total_net_input_bytes
net.output — stats.total_net_output_bytes
latest_fork — stats.latest_fork_usec
clients.connected — clients.connected_clients
clients.blocked — clients.blocked_clients
clients.max_input_buffer — clients.client_recent_max_input_buffer
clients.max_output_buffer — clients.client_recent_max_output_buffer
slaves.connected — replication.connected_slaves
replication.offset — replication.master_repl_offset
replication.backlog_first_byte_offset — replication.repl_backlog_first_byte_offset
role role ∈ primary, replica; value 1 replication.role = master / slave (a point whose value isn’t 1 is skipped silently, not counted as dropped)
uptime — server.uptime_in_seconds
rdb.changes_since_last_save — persistence.rdb_changes_since_last_save
db.keys / db.expires / db.avg_ttl db composite keyspace.db<N> → keys / expires / avg_ttl
cmd.calls / cmd.usec cmd composite commandstats.cmdstat_<cmd> → calls / usec
resource redis.version — server.redis_version, plus capabilities and CVE scanning for Redis
resource valkey.version — server.valkey_version once the connection is detected as Valkey, plus capabilities and CVE scanning for Valkey

What works. Memory history and forecasting, anomaly detection (INFO-based detectors, limited to the fields the collector pushes), health and instance-down webhooks, and commandstats all run against pushed data the same as a polled connection.

CVE scanning. The Security page, the betterdb_cve_* metrics and new-CVE webhooks work for OTLP push connections. Instead of reading INFO, the scan uses the engine version the collector reports. A Redis is matched on the redis.version resource attribute, which redisreceiver sets by default. Valkey reports a fixed redis_version of 7.2.4 for client compatibility, so a connection detected as Valkey (db.system.name=valkey, or valkey.* metric names) is matched only on a valkey.version resource attribute. Set both with a resource processor, and update value when you upgrade:

processors:
  resource/valkey:
    attributes:
      - key: valkey.version
        value: '8.1.1'
        action: upsert
      - key: db.system.name
        value: valkey
        action: upsert

service:
  pipelines:
    metrics:
      receivers: [redis]
      processors: [resource/valkey]
      exporters: [otlphttp/betterdb]

A connection is scanned as soon as its version first arrives or changes, and again every 6 hours. Until a version arrives nothing is scanned or saved, and the Security page names the missing attribute. Modules are never pushed, so module advisories are reported as unverifiable. Without either Valkey signal, a Valkey instance is scanned as Redis 7.2.4. Pushed versions live in memory, so after a restart the last stored scan stays on screen until the next push.

What doesn’t. Every view that needs a live command against the instance stays unavailable: slow log, clients, latency, key analytics, cluster, audit trail, vector search, and the other live-only pages. They show: “Not available for OTLP-ingested connections — this view needs a live connection.” Migration is a related but separate case — it’s blocked at the source/target picker instead, with its own message (“One or more selected instances only pushes OTLP metrics. Migration needs a live connection to both instances.”), since an OTLP-ingested connection is never a valid migration source or target.

Prometheus. OTLP-push connections are exported at /api/prometheus/metrics like polled ones, with one difference: a series exists only for a field the collector actually pushed. A field that was never pushed, or stopped being pushed, has no series rather than a 0. Cluster, slot-stats, raw slow log, ACL and client-analytics series are never produced for these connections, and webhooks or compliance alerts that depend on an unpushed field (for example maxclients or maxmemory_policy) don’t fire. Ingest itself is counted by betterdb_otlp_metric_points_accepted_total and betterdb_otlp_metric_points_dropped_total{reason} (full export profile only).

Drop reasons. A response’s partialSuccess.errorMessage summarises rejected points by reason, for example unknown_instance=12 unmapped_metric=3:

Reason Meaning
unidentified The resource has no service.instance.id, or one that can’t be parsed as host:port/host-port, and there’s no server.address/server.port fallback either — so no instance could be resolved.
unknown_instance The resolved host:port isn’t a registered connection. The instance is listed under “discovered via OTLP” in the connection switcher unless discovery is off.
already_polled The resolved host:port is a connection BetterDB already polls directly (not registered as OTLP push).
unsupported_type The point is a histogram, exponential histogram, or summary.
unsupported_temporality The point is a sum with delta (not cumulative) temporality.
unmapped_metric The metric name or its attributes don’t match anything in the table above.
invalid_value The point has no value, or its value isn’t a finite number.
cardinality_limit The point would add a new keyspace or commandstats entry beyond the per-connection cap (256 databases, 1024 commands). Entries whose values have all gone stale free their slot.

Staleness. OTEL_METRICS_STALE_AFTER_MS (default 5 minutes) sets how long a pushed point stays valid. Set it well above the exporter’s push interval (at least 2x); a window shorter than the interval makes health flap down and up between pushes, so values under 60000 are rejected at startup. Once every field has gone stale with no fresh push (a keyspace or commandstats entry only counts while its keys or calls value is fresh), the connection shows as disconnected and health fires an instance-down event. Staleness counts from when Monitor received a point, so a collector whose clock runs behind does not age data out early. A point’s own timestamp only decides which of two values for the same field is newer (a point stamped ahead of Monitor’s clock is treated as received now). After a BetterDB restart, a connection stays disconnected until the next push arrives — nothing is lost, since prior history is kept in storage.

Memory snapshots. A pushed instance gets a memory snapshot whenever used_memory is present. When the exporter didn’t push used_memory_rss, used_memory_peak, mem_fragmentation_ratio, allocator_frag_ratio or the io-thread read/write counters, that field is stored empty (null in the API) rather than as 0, so it never reads as a real zero. Charts show a gap for it, forecasting skips those samples, the migration baseline averages only the samples that carry a fragmentation ratio (and shows — when none do), and the Prometheus export omits the series. maxmemory, ops/sec and CPU still fall back to 0 when absent. redisreceiver never emits allocator_frag_ratio or the io-thread counters, and of the memory fields only valkey.memory.used is confirmed for Valkey Admin’s exporter.

Commandstats history. A command is recorded whenever its cmd.calls value moves. When cmd.usec wasn’t pushed for it, the sample keeps its call counts and stores usecTotal, usecPerCall and usecDelta as null, so the latency chart shows a gap. rejected_calls and failed_calls have no OTLP mapping, so they are always null for a pushed instance (they are left out of the /metrics/commandstats/summary response). Polled connections are unchanged: a server too old to report rejected_calls and failed_calls still records 0.

Auth. Ingestion is gated the same way as trace ingestion: OTEL_INGEST_ENABLED and OTEL_INGEST_TOKEN (mandatory in cloud mode).

Metrics export

Set OTEL_EXPORTER_OTLP_ENDPOINT and Monitor mirrors its Prometheus registry to OTLP metrics, pushing to ${endpoint}/v1/metrics on an interval as service betterdb-monitor.

A caveat worth knowing: counters and gauges are mirrored, but histograms and summaries are skipped because they do not map cleanly onto the OTLP instruments here. So the OTLP mirror is a subset. For the complete set, including histograms and every betterdb_* family, scrape the Prometheus endpoint at /api/prometheus/metrics (see Prometheus Integration and the full metrics reference).

The export reads the same registry as /api/prometheus/metrics. In both export modes, a series that leaves that registry, such as a stale or removed connection’s series or the previous role after a failover, stops being exported once it has gone unobserved for the Prometheus staleness bound (see PROMETHEUS_STALENESS_MS), or for two export intervals if that is longer. If the series comes back, it is exported again from the next push.

Tune the push interval with OTEL_METRICS_EXPORT_INTERVAL_MS (default 15000).

Export modes

OTEL_METRICS_EXPORT_MODE picks how metrics are named and grouped. It is read once at startup.

  • mirror (default) keeps the Prometheus names (betterdb_memory_used_bytes) on one betterdb-monitor resource, with a connection attribute on every point.
  • semconv sends one OTLP resource per monitored node, named after the OpenTelemetry Collector redisreceiver catalogue with a valkey. prefix, so dashboards built for that receiver work unchanged.

Any other value logs a warning and uses mirror. METRICS_EXPORT_PROFILE applies in both modes.

Resource attributes in semconv mode

Attribute Value
service.name valkey or redis (valkey until the first poll detects the type)
service.instance.id host:port
db.system.name valkey or redis (omitted until detected)
server.address host
server.port port
valkey.version server version (omitted until detected)

Monitor process metrics (betterdb_process_*, betterdb_nodejs_*) stay on the betterdb-monitor resource. Families with an OpenTelemetry process, V8 or Node.js convention are renamed (see Monitor process below); the rest keep their Prometheus names. A point whose connection is not registered also lands there and keeps its connection attribute. Agent connections have no host:port of their own, so their points land there too.

After a connection is removed, its points are no longer exported: they are dropped, not moved to the betterdb-monitor resource.

Monitor sends one OTLP request per node plus one for the monitor each interval, at most eight at a time. Each push reports its result before the export interval runs out. Requests still running at that point continue, and if one of them fails, the failure is reported with the next push.

Metric names in semconv mode

Receiver-aligned

Prometheus family Semconv metric Unit
betterdb_memory_used_bytes valkey.memory.used By
betterdb_memory_used_rss_bytes valkey.memory.rss By
betterdb_memory_used_peak_bytes valkey.memory.peak By
betterdb_memory_max_bytes valkey.maxmemory By
betterdb_memory_fragmentation_ratio valkey.memory.fragmentation_ratio 1
betterdb_cpu_sys_seconds_total valkey.cpu.time s
betterdb_cpu_user_seconds_total valkey.cpu.time s
betterdb_connected_clients valkey.clients.connected {client}
betterdb_blocked_clients valkey.clients.blocked {client}
betterdb_commands_processed_total valkey.commands.processed {command}
betterdb_connections_received_total valkey.connections.received {connection}
betterdb_instantaneous_ops_per_sec valkey.commands {ops}/s
betterdb_keyspace_hits_total valkey.keyspace.hits {hit}
betterdb_keyspace_misses_total valkey.keyspace.misses {miss}
betterdb_evicted_keys_total valkey.keys.evicted {key}
betterdb_expired_keys_total valkey.keys.expired {event}
betterdb_db_keys valkey.db.keys {key}
betterdb_db_keys_expiring valkey.db.expires {key}
betterdb_db_avg_ttl_seconds valkey.db.avg_ttl ms
betterdb_commandstats_calls_total valkey.cmd.calls {call}
betterdb_connected_slaves valkey.slaves.connected {replica}
betterdb_replication_offset valkey.replication.offset By
betterdb_uptime_in_seconds valkey.uptime s
betterdb_rdb_changes_since_last_save valkey.rdb.changes_since_last_save {change}
betterdb_cluster_enabled valkey.cluster.cluster_enabled 1
betterdb_cluster_known_nodes valkey.cluster.known_nodes {node}
betterdb_cluster_slots_assigned valkey.cluster.slots_assigned {slot}
betterdb_cluster_slots_ok valkey.cluster.slots_ok {slot}
betterdb_cluster_slots_fail valkey.cluster.slots_fail {slot}
betterdb_cluster_slots_pfail valkey.cluster.slots_pfail {slot}
betterdb_instance_info valkey.role {role}

BetterDB-only

Prometheus family Semconv metric Unit
betterdb_memory_fragmentation_bytes betterdb.memory.fragmentation By
betterdb_tracking_clients betterdb.clients.tracking {client}
betterdb_instantaneous_input_kbps betterdb.net.input_rate KiBy/s
betterdb_instantaneous_output_kbps betterdb.net.output_rate KiBy/s
betterdb_pubsub_channels betterdb.pubsub.channels {channel}
betterdb_pubsub_patterns betterdb.pubsub.patterns {pattern}
betterdb_key_pattern_memory_bytes betterdb.key_pattern.memory.total By
betterdb_key_pattern_avg_memory_bytes betterdb.key_pattern.memory.avg By
betterdb_key_pattern_max_memory_bytes betterdb.key_pattern.memory.max By
betterdb_key_pattern_keys betterdb.key_pattern.keys {key}
betterdb_key_memory_bytes betterdb.key.memory By
betterdb_key_cardinality betterdb.key.cardinality {element}
betterdb_keysize_bucket_keys betterdb.keysizes.bucket.keys {key}
betterdb_repl_output_buffer_ratio betterdb.replication.output_buffer_ratio 1
betterdb_master_link_up betterdb.replication.master_link_up 1
betterdb_master_last_io_seconds_ago betterdb.replication.master_last_io_age s
betterdb_rdb_last_save_timestamp_seconds betterdb.rdb.last_save_time s
betterdb_rdb_last_bgsave_ok betterdb.rdb.last_bgsave_ok 1
betterdb_aof_enabled betterdb.aof.enabled 1
betterdb_aof_last_bgrewrite_ok betterdb.aof.last_bgrewrite_ok 1
betterdb_cluster_size betterdb.cluster.size {node}
betterdb_cluster_stats_messages_crc_mismatch betterdb.cluster.messages_crc_mismatch {message}
betterdb_cluster_slot_keys betterdb.cluster.slot.keys {key}
betterdb_cluster_slot_expires betterdb.cluster.slot.expires {key}
betterdb_cluster_slot_reads_total betterdb.cluster.slot.reads {read}
betterdb_cluster_slot_writes_total betterdb.cluster.slot.writes {write}
betterdb_commandstats_latency_us betterdb.cmd.latency_avg us
betterdb_slowlog_length betterdb.slowlog.length {entry}
betterdb_slowlog_last_id betterdb.slowlog.last_id {id}
betterdb_slowlog_pattern_count betterdb.slowlog.pattern.count {entry}
betterdb_slowlog_pattern_avg_duration_us betterdb.slowlog.pattern.duration_avg us
betterdb_slowlog_pattern_percentage betterdb.slowlog.pattern.share %
betterdb_commandlog_large_request betterdb.commandlog.large_request {entry}
betterdb_commandlog_large_reply betterdb.commandlog.large_reply {entry}
betterdb_commandlog_large_request_by_pattern betterdb.commandlog.large_request_by_pattern {entry}
betterdb_commandlog_large_reply_by_pattern betterdb.commandlog.large_reply_by_pattern {entry}
betterdb_acl_denied betterdb.acl.denied {event}
betterdb_acl_denied_by_reason betterdb.acl.denied_by_reason {event}
betterdb_acl_denied_by_user betterdb.acl.denied_by_user {event}
betterdb_client_connections_current betterdb.client.connections {connection}
betterdb_client_connections_peak betterdb.client.connections_peak {connection}
betterdb_client_connections_by_name betterdb.client.connections_by_name {connection}
betterdb_client_connections_by_user betterdb.client.connections_by_user {connection}
betterdb_vector_index_docs betterdb.vector_index.docs {document}
betterdb_vector_index_memory_bytes betterdb.vector_index.memory By
betterdb_vector_index_indexing_failures betterdb.vector_index.indexing_failures {failure}
betterdb_vector_index_percent_indexed betterdb.vector_index.indexed %
betterdb_inference_bucket_p50_us betterdb.inference.bucket.latency us
betterdb_inference_bucket_p95_us betterdb.inference.bucket.latency us
betterdb_inference_bucket_p99_us betterdb.inference.bucket.latency us
betterdb_inference_unhealthy betterdb.inference.bucket.unhealthy 1
betterdb_inference_sla_breach betterdb.inference.sla_breach 1
betterdb_poll_stale betterdb.poll.stale 1
betterdb_polls_total betterdb.poll.count {poll}
betterdb_anomaly_events_total betterdb.anomaly.events {event}
betterdb_anomaly_events_current betterdb.anomaly.active {event}
betterdb_anomaly_by_severity betterdb.anomaly.by_severity {event}
betterdb_anomaly_by_metric betterdb.anomaly.by_metric {event}
betterdb_correlated_groups_total betterdb.anomaly.correlated_groups {group}
betterdb_correlated_groups_by_severity betterdb.anomaly.correlated_groups_by_severity {group}
betterdb_correlated_groups_by_pattern betterdb.anomaly.correlated_groups_by_pattern {group}
betterdb_anomaly_buffer_ready betterdb.anomaly.buffer.ready 1
betterdb_anomaly_buffer_mean betterdb.anomaly.buffer.mean —
betterdb_anomaly_buffer_stddev betterdb.anomaly.buffer.stddev —
betterdb_metric_forecast_time_to_limit_seconds betterdb.forecast.time_to_limit s
betterdb_cve_findings betterdb.cve.findings {finding}
betterdb_cve_kev betterdb.cve.kev {finding}
betterdb_cve_dataset_stale betterdb.cve.dataset_stale 1
betterdb_otlp_metric_points_accepted_total betterdb.otlp.metric_points.accepted {point}
betterdb_otlp_metric_points_dropped_total betterdb.otlp.metric_points.dropped {point}

Monitor process

Prometheus family Semconv metric Unit
betterdb_process_cpu_user_seconds_total process.cpu.time{cpu.mode=user} s
betterdb_process_cpu_system_seconds_total process.cpu.time{cpu.mode=system} s
betterdb_process_resident_memory_bytes process.memory.usage By
betterdb_process_virtual_memory_bytes (Linux) process.memory.virtual By
betterdb_process_open_fds (Linux) process.unix.file_descriptor.count {file_descriptor}
betterdb_process_start_time_seconds process.uptime s
betterdb_nodejs_heap_space_size_used_bytes v8js.memory.heap.used By
betterdb_nodejs_heap_space_size_total_bytes v8js.memory.heap.space.size By
betterdb_nodejs_heap_space_size_available_bytes v8js.memory.heap.space.available_size By
betterdb_nodejs_eventloop_lag_{min,max,mean,stddev,p50,p90,p99}_seconds nodejs.eventloop.delay.{min,max,mean,stddev,p50,p90,p99} s

The heap families carry v8js.heap.space.name (old → old_space), and process.uptime is the time since the start timestamp. Other process families, such as betterdb_process_cpu_seconds_total (the sum of the two CPU modes), betterdb_process_max_fds, betterdb_nodejs_eventloop_lag_seconds and the heap totals, keep their Prometheus names. betterdb_nodejs_gc_duration_seconds is a histogram, so it is not exported.

Conversions: db values drop the db prefix (db0 → 0), valkey.db.avg_ttl is in milliseconds, the two CPU families become valkey.cpu.time with state=sys|user, betterdb_instance_info becomes valkey.role{role=primary|replica} (the version moves to the resource), command is renamed cmd, and the three inference percentile families become betterdb.inference.bucket.latency{percentile}. betterdb_keyspace_keys and betterdb_keyspace_keys_expiring are not exported in this mode because they are sums of valkey.db.*. betterdb_poll_duration_seconds is a histogram, so it is not exported in either mode.

Event export

When an OTLP endpoint is configured, Monitor also emits discrete monitoring events as OTLP log records to ${endpoint}/v1/logs (logger betterdb-events). These are the same events that drive webhooks: instance up/down, cluster failover, cluster bus corruption, and compliance alerts.

Environment variables

Variable Default Description
OTEL_INGEST_ENABLED true Enable the OTLP receivers: /v1/traces (traces) and /v1/external/metrics (metrics ingestion).
OTEL_INGEST_TOKEN unset Bearer token required to post traces or push metrics. Required in cloud mode.
OTEL_METRICS_STALE_AFTER_MS 300000 Age after which a pushed metric is ignored; min 60000. Must exceed the exporter push interval (at least 2x).
OTLP_DISCOVER_INSTANCES true Record pushes from unregistered host:ports and list them in the connection switcher. On unless set to false, 0, no or off.
OTEL_EXPORTER_OTLP_ENDPOINT unset Base URL of your OTLP/HTTP collector. Setting it enables the metrics and event exports; /v1/metrics and /v1/logs are appended automatically.
OTEL_TELEMETRY_ENABLED true Set false to disable the metrics and event exports even when an endpoint is set.
OTEL_EVENTS_ENABLED true Set false to keep the metrics mirror but disable just the event/log export (/v1/logs). Needed when the endpoint is a metrics-only receiver (e.g. Prometheus’s native OTLP receiver), which would 404 every event. Off on false, 0, no or off.
OTEL_METRICS_EXPORT_INTERVAL_MS 15000 Metrics mirror push interval in milliseconds (minimum 1000).
OTEL_METRICS_EXPORT_MODE mirror mirror or semconv. See Export modes.

Kubernetes

The install chart wires the export endpoint through an env var. See Kubernetes install for a full manifest:

env:
  - name: OTEL_EXPORTER_OTLP_ENDPOINT
    value: 'http://otel-collector.observability.svc.cluster.local:4318'

OpenTelemetry Collector

A tested Collector config ships in deploy/observability/collector/:

  • otel-collector.yaml — the base config. It receives OTLP on 0.0.0.0:4317 (grpc) and 0.0.0.0:4318 (http) — the same endpoint you point OTEL_EXPORTER_OTLP_ENDPOINT at — batches and memory-limits the stream, and re-exposes it as a Prometheus scrape target on :8889.
  • otel-collector.fanout.yaml — the same intake, additionally exporting to a second OTLP backend of your choice via the SECONDARY_OTLP_ENDPOINT environment variable (for example a hosted metrics backend), alongside the same Prometheus exporter.

Both configs are plain YAML mounted into the Collector container — editing them doesn’t require rebuilding an image.

Monitor pushes its OTLP mirror on OTEL_METRICS_EXPORT_INTERVAL_MS (default 15000ms). Point Prometheus at the Collector’s exporter on :8889 rather than at Monitor directly if you want the Collector in the path.

connection is a datapoint attribute, not a resource attribute, so the Prometheus exporter turns it into a connection label on every metric regardless of configuration. The exporter’s resource_to_telemetry_conversion setting covers the resource attributes instead — Monitor sets exactly one, service.name, which the setting promotes to a service_name label. That is why the dashboard pack, which is templated on connection, works unmodified whether Grafana points at Monitor’s own /api/prometheus/metrics or at the Collector’s :8889.

See the observability pack README for import instructions and a full docker-compose demo of Monitor, the Collector, Prometheus, and Grafana running together.

Summary

  • Traces: Monitor receives OTLP traces at /v1/traces (JSON and protobuf). It does not export its own spans.
  • Metrics ingestion: Monitor receives OTLP metrics at /v1/external/metrics for registered OTLP-push connections, driving memory history, forecasting, anomaly detection, health and the Prometheus export for instances it doesn’t dial directly.
  • Metrics: Prometheus-first at /api/prometheus/metrics; opt-in OTLP mirror of counters and gauges when an endpoint is set (histograms via Prometheus only).
  • Events: opt-in OTLP logs for the same events that trigger webhooks.