Webhook Notifications

BetterDB can send HTTP POST requests to your endpoints when monitoring events occur. Webhooks provide real-time notifications for critical events, enabling integration with external alerting systems, incident management platforms, and custom automation workflows.

Table of Contents

Quick Start

Via Web UI

  1. Navigate to Settings → Webhooks in the BetterDB Monitor interface
  2. Click Create Webhook
  3. Enter your endpoint URL (e.g., https://api.example.com/webhooks/betterdb)
  4. Select events to subscribe to
  5. (Optional) Add a secret for HMAC signature verification
  6. (Optional) Configure custom headers and retry policy
  7. Click Test to verify connectivity
  8. Save the webhook

Via API 

curl -b cookies.txt -c cookies.txt -X POST http://localhost:3001/api/webhooks \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Production Alerts",
    "url": "https://api.example.com/webhooks/betterdb",
    "secret": "your-secret-key",
    "events": ["instance.down", "instance.up", "memory.critical"],
    "enabled": true
  }'

Event Types

BetterDB webhooks are tiered by license level. Each tier includes events from lower tiers.

Community Tier (Free)

All self-hosted BetterDB installations have access to these events:

Event Description Trigger Condition
instance.down Database unreachable Connection/ping failure detected
instance.up Database recovered Connection restored after failure
memory.critical Memory usage critical Memory exceeds 90% of maxmemory
connection.critical Connection limit critical Connections exceed 90% of maxclients
client.blocked Authentication failure ACL log entry with reason auth
writes.rejected Server is refusing writes BGSAVE or AOF write failed, or errorstat_MISCONF / OOM / READONLY / NOREPLICAS rose since the last poll
writes.recovered Server accepts writes again No new write-rejection errors for 60s, memory below maxmemory for OOM, and persistence reported ok for BGSAVE/AOF causes

Pro Tier

Advanced monitoring events for anomaly detection and performance tracking:

Event Description Trigger Condition
slowlog.threshold Slow query detected Slowlog count exceeds configured threshold
replication.lag Replication lag detected Replica lag exceeds acceptable threshold
replication.divergence Replica falling behind A replica’s byte gap to the primary keeps growing, stays far behind, or stops draining
replication.recovered Replica divergence closed A diverged replica is back under the alert thresholds, or a divergence left open by a restart was resolved
cluster.failover Cluster failover occurred Cluster state changes or slot failures
anomaly.detected Anomaly detected Z-score analysis detects unusual patterns. Sent when the event is first saved, so it waits for storage if a save has to be retried. command_p99, retry_amplification and replication_divergence are excluded; they send latency.regression.detected, retry_amplification.detected and replication.divergence instead
anomaly.sequence.detected Anomaly sequence matched A sequence rule matched inside an incident. Sent once when the match is recorded; nothing is sent on recovery. A compound rule that replaces an earlier match sends a second alert with the more severe diagnosis
latency.spike Latency spike detected Command latency spikes above baseline
connection.spike Connection spike detected Connection count spikes above baseline
cve.critical_detected New critical CVEs CVE scan finds new critical findings vs previous scan
scaling_readiness.low Scaling readiness low Scaling readiness score dropped to or below the connection’s alert threshold
kv_cache.hit_rate_low KV cache hit rate low An LMCache engine’s token hit rate over the last 15 minutes dropped to or below the connection’s threshold
kv_cache.eviction_risk KV cache eviction risk LMCache keys cannot be evicted under a volatile-* policy, or Valkey is evicting while LMCache holds most of its memory
retry_amplification.detected Retry amplification detected Load per client keeps rising, or steps up and holds, together with error replies or connection churn while the client count stays flat

Enterprise Tier

Compliance and audit events for regulated environments:

Event Description Trigger Condition
compliance.alert Compliance policy violation Memory high with noeviction policy (data loss risk)
audit.policy.violation ACL policy violation Command/key access denied by ACL
acl.violation ACL access violation Runtime ACL access denied
acl.modified ACL configuration changed User added/removed or permissions changed
config.changed Database configuration changed CONFIG SET command executed
cve.kev_detected New KEV-exploited CVEs CVE scan finds new CISA KEV findings vs previous scan

Payload Format

Set payloadFormat per webhook (generic default, slack, or discord). generic sends the raw BetterDB event JSON below; slack sends Block Kit blocks and discord sends an embed with the same fields (instance, metric, value/baseline, link back to /anomalies or /dashboard). For anomaly.detected and anomaly.sequence.detected the link opens the incident that holds the event. A message too long for the title is shown in full below it. HMAC headers are still sent for all formats; Slack/Discord ignore them.

Getting the webhook URL: Slack — create an app and enable an Incoming Webhook in its configuration; Discord — Channel Settings → Integrations → Webhooks → New Webhook. In the BetterDB form, pick the matching payload format (it is auto-suggested from the URL).

The “View in BetterDB” button (Slack) / embed link (Discord) points at your FRONTEND_URL (e.g. https://monitor.example.com). When FRONTEND_URL is unset, messages render without the link — everything else is unchanged. For anomaly.detected and anomaly.sequence.detected the link opens the incident that holds the event, looked up again each time it is opened, so it keeps working when the incident is regrouped.

// generic (default)
{ "id": "...", "event": "anomaly.detected", "timestamp": 1706457600000,
  "instance": { "host": "valkey.example.com", "port": 6379 },
  "data": { "metricType": "latency", "value": 42, "baseline": 10 } }

A memory.critical event renders exactly as follows (FRONTEND_URL=https://monitor.example.com):

// slack (payloadFormat: "slack") — Block Kit
{
  "text": "Memory usage critical: 92.5% (threshold: 90%)",
  "blocks": [
    {
      "type": "section",
      "text": {
        "type": "mrkdwn",
        "text": "*Memory usage critical: 92.5% (threshold: 90%)*"
      }
    },
    {
      "type": "section",
      "fields": [
        { "type": "mrkdwn", "text": "*Event:*\nmemory.critical" },
        { "type": "mrkdwn", "text": "*Instance:*\nvalkey.example.com:6379" },
        { "type": "mrkdwn", "text": "*Metric:*\nmemory_used_percent" },
        { "type": "mrkdwn", "text": "*Value / baseline:*\n92.5 / 90" }
      ]
    },
    {
      "type": "context",
      "elements": [
        { "type": "mrkdwn", "text": "<!date^1706457600^{date_short} {time}|alert time>" }
      ]
    },
    {
      "type": "actions",
      "elements": [
        {
          "type": "button",
          "text": { "type": "plain_text", "text": "View in BetterDB" },
          "url": "https://monitor.example.com/dashboard"
        }
      ]
    }
  ]
}

// discord (payloadFormat: "discord") — embed
{
  "content": "Memory usage critical: 92.5% (threshold: 90%)",
  "embeds": [
    {
      "title": "Memory usage critical: 92.5% (threshold: 90%)",
      "color": 4088797,
      "fields": [
        { "name": "Event", "value": "memory.critical", "inline": true },
        { "name": "Instance", "value": "valkey.example.com:6379", "inline": true },
        { "name": "Metric", "value": "memory_used_percent", "inline": true },
        { "name": "Value / baseline", "value": "92.5 / 90", "inline": true }
      ],
      "timestamp": "2024-01-28T16:00:00.000Z",
      "url": "https://monitor.example.com/dashboard",
      "footer": { "text": "BetterDB Monitor" }
    }
  ]
}

Use POST /webhooks/:id/test to preview: the response includes payloadFormat and the exact renderedPayload that would be sent — including on failure, so you can see what a failing Slack/Discord hook would have received.

All webhooks send JSON payloads with this structure:

{
  "id": "550e8400-e29b-41d4-a716-446655440000",
  "event": "instance.down",
  "timestamp": 1706457600000,
  "instance": {
    "host": "valkey.example.com",
    "port": 6379
  },
  "data": {
    // Event-specific payload (see below)
  }
}

HTTP Headers

BetterDB sends these headers with every webhook request:

Content-Type: application/json
User-Agent: BetterDB-Monitor/1.0
X-Webhook-Signature: <hmac-sha256-signature>
X-Webhook-Timestamp: <unix-timestamp-ms>
X-Webhook-Id: <webhook-id>
X-Webhook-Delivery-Id: <delivery-id>
X-Webhook-Event: <event-type>

Event-Specific Payloads

instance.down / instance.up 

{
  "message": "Database instance unreachable: Connection refused",
  "reason": "Connection refused",
  "detectedAt": "2024-01-28T12:00:00.000Z"
}

memory.critical 

{
  "metric": "memory_used_percent",
  "value": 92.5,
  "threshold": 90,
  "maxmemory": 1073741824,
  "usedMemory": 993001472,
  "message": "Memory usage critical: 92.5% (threshold: 90%)"
}

connection.critical 

{
  "metric": "connected_clients_percent",
  "value": 95.2,
  "threshold": 90,
  "maxClients": 10000,
  "connectedClients": 9520,
  "message": "Connection usage critical: 95.2% (threshold: 90%)"
}

client.blocked 

{
  "reason": "auth",
  "username": "app_user",
  "clientInfo": "192.168.1.100:54321",
  "count": 5,
  "timestamp": 1706457600,
  "message": "Client blocked: authentication failure by app_user@192.168.1.100:54321 (count: 5)"
}

writes.rejected / writes.recovered

Fired when the server starts and stops refusing writes. None of these conditions moves memory, CPU or latency, so they are read from the persistence and errorstats INFO sections the metrics poll already fetches.

writes.rejected opens with severity: "warning" as soon as persistence fails (rdb_last_bgsave_status:err, which makes the server reply MISCONF to writes under the default stop-writes-on-bgsave-error yes, or aof_last_write_status:err), before any client has been refused. When it opens because clients already received errors, severity is "critical" and rejectedSinceLastPoll holds the per-code counts. OOM, READONLY and NOREPLICAS leave no persistent “still broken” flag, so one quiet poll is not proof of recovery. writes.recovered fires when:

  • 60 seconds pass with no new rejections and, for maxmemory_reached, used_memory minus mem_not_counted_for_evict is below maxmemory (matching Redis’ own check).
  • Only edges with rdb_bgsave_failed or aof_write_failed also require INFO persistence to report the statuses as ok; a missing persistence section keeps those open. The RDB status is ignored when a failed BGSAVE doesn’t stop writes (stop-writes-on-bgsave-error no or no save points).
  • Errorstats-only edges (maxmemory_reached, write_to_replica, min_replicas_not_met, persistence_failed) don’t depend on persistence, so they also recover on hosts that hide the persistence section.
Cause Meaning
rdb_bgsave_failed Last BGSAVE failed (disk full, fork ENOMEM, permissions) — clients get MISCONF. Only reported when stop-writes-on-bgsave-error yes and save points are configured; if CONFIG GET is blocked it is assumed to be on
aof_write_failed Last AOF write failed — clients get MISCONF
maxmemory_reached OOM command not allowed (e.g. noeviction at maxmemory)
write_to_replica READONLY — clients writing to a replica, typically stale topology after failover
min_replicas_not_met NOREPLICAS — min-replicas-to-write not satisfied
persistence_failed MISCONF seen with no blocking persistence failure visible (status cleared, persistence section absent, or BGSAVE failure not configured to stop writes)

READONLY from a node that reports role:slave is expected (probes, misrouted clients) and never opens an edge; it only alerts on a node that reports role:master.

On the first poll error counters are only baselined, so historical counts never alert; a persistence failure is still detected immediately. A counter drop (restart, CONFIG RESETSTAT) re-baselines instead of firing. errorstats needs Redis 6.2+ or Valkey; on older servers only the persistence causes are detected.

If clients start receiving errors (or a new cause appears) while an edge is open, writes.rejected is sent once more with escalated: true. Its severity is "critical" when clients are receiving errors; a new persistence cause with no client errors yet keeps "warning". It is not re-sent while the edge stays critical with the same causes, and once an edge has gone critical every later escalation reports "critical".

Field Type Description
causes string[] Active causes on this poll (see table above); empty on writes.recovered
severity "warning" | "critical" critical once clients have received a rejection error on this edge
rejectedSinceLastPoll object Rejection errors returned to clients since the last poll, by code (MISCONF, OOM, READONLY, NOREPLICAS)
rejectingForMs number Time since the edge opened; on writes.recovered, how long writes were rejected
escalated boolean Present and true only on a repeated writes.rejected for an edge that is already open
message string Human-readable summary

Escalation of an open edge, after clients start receiving MISCONF:

{
  "causes": ["rdb_bgsave_failed"],
  "severity": "critical",
  "rejectedSinceLastPoll": { "MISCONF": 250 },
  "rejectingForMs": 5000,
  "escalated": true,
  "message": "Writes are being rejected (rdb_bgsave_failed) and clients are receiving errors",
  "timestamp": 1706457600000,
  "instance": { "host": "localhost", "port": 6379 }
}

{
  "causes": [],
  "severity": "warning",
  "rejectedSinceLastPoll": {},
  "rejectingForMs": 95000,
  "message": "Writes accepted again after 95s",
  "timestamp": 1706457695000,
  "instance": { "host": "localhost", "port": 6379 }
}

slowlog.threshold (Pro) 

{
  "slowlogCount": 150,
  "threshold": 100,
  "message": "Slowlog count (150) exceeds threshold (100)",
  "timestamp": 1706457600000
}

replication.lag (Pro) 

{
  "lagSeconds": 45,
  "threshold": 30,
  "masterLinkStatus": "up",
  "message": "Replication lag (45s) exceeds threshold (30s)",
  "timestamp": 1706457600000
}

replication.divergence (Pro)

Sent when a divergence opens or escalates, once the event has been saved. If storage is down the webhook waits for the retry, and is skipped if the event closes first. reason is one of growth, ceiling, not_catching_up, not_applying or not_acking. not_applying means writes have stopped and the gap is not draining; a window counts as stopped when the replication stream stays under 4 KiB/s, so PINGs count as idle. secondsBehind is null when writes have stopped.

When the primary and the stalled replica are both monitored, a webhook subscribed to both replication.lag and replication.divergence gets one alert when the replica stops acknowledging writes, whichever is delivered first:

  • Divergence first. Once the webhook has received a not_acking divergence for the open stall, the replica’s replication.lag alert is skipped. A divergence with another reason does not skip it. The skipped alert stays latched, so it is not sent until lag drops below the recovery level and crosses the threshold again, even if the stall recovers in between.
  • Lag first. If the webhook’s lag alert is still active when the stall opens with not_acking, that divergence is skipped, and so is the stall’s replication.recovered. A lag alert covers one stall only: an escalation to critical is sent (followed by replication.recovered), and so is any later stall.

An alert only counts once its delivery succeeds, so a failed or dropped delivery never suppresses the other one. Divergences with other reasons are always sent. The anomaly event is still recorded either way.

replicaView is what the replica itself reported when the monitor could read it: gapBytes from its slave_repl_offset, linkStatus from master_link_status, and applyBacklogBytes, the data it has received but not applied (null when the server does not report slave_read_repl_offset). It is null when the replica could not be read. A divergence is not sent when the replica reports its link is up and it is less than 1 MiB behind, except a not_acking one: that is still sent, and its message says the replica’s acknowledgements are not reaching the primary. See Replica cross-check.

Detection works on windows of at least 30 seconds, or three poll intervals if that’s longer. If polling stops for longer than a window and more than twice the previous gap between polls, the monitor treats it as an outage: every streak restarts, so an alert can take several windows to fire after polling resumes. A divergence that was already open stays open. Polling that stays steadily slower than a window is not an outage, and detection continues on longer windows.

{
  "replica": "10.0.0.2:6379",
  "level": "critical",
  "reason": "ceiling",
  "gapBytes": 629145600,
  "secondsBehind": 600,
  "bytesPerSec": 1048576,
  "slopeBytesPerWindow": -1048576,
  "windowCount": 2,
  "correlatedSignals": ["memory_used"],
  "replicaView": { "gapBytes": 629145600, "linkStatus": "up", "applyBacklogBytes": 524288000 },
  "message": "CRITICAL: Replica 10.0.0.2:6379 is 600 MB (~600s at the current write rate) behind the primary; it has stayed at least 120s behind for 2 windows. The replica confirms it is 600 MB behind; it has received 500 MB it has not applied yet, so it is slow to apply.",
  "timestamp": 1706457600000,
  "instance": { "host": "10.0.0.1", "port": 6379 },
  "connectionId": "conn-1"
}

replication.recovered (Pro)

reason is caught_up when the replica stays under the alert thresholds for two windows: its gap is below 1 MiB, or less than 5 seconds behind at the current write rate. gapBytes can therefore be above zero, and above 1 MiB on a fast primary. It is resolved_after_restart when the monitor closes a divergence the previous run left open and the replica has stayed under the alert thresholds for two windows; gapBytes is then null. A replica that is still behind but below the alert thresholds keeps its event open.

No replication.recovered is sent when the replica was not seen to catch up: when an alerted replica goes offline or drops out of the primary’s replica list, when the node stops being a primary, or when a replica with a restart-orphaned event is gone. The anomaly event closes silently, because a dropped replica is usually worse off than one that was behind. Treat a missing recovery as unknown, and watch replication.lag or the replica’s connection state if you need to page on it.

If the replica alerts again, the old event closes with no replication.recovered, since the new replication.divergence replaces it. A restart sends at most one per replica, and none when the event was already resolved elsewhere. Removing a connection closes its divergence events without sending replication.recovered.

{
  "replica": "10.0.0.2:6379",
  "reason": "caught_up",
  "gapBytes": 0,
  "message": "Replica 10.0.0.2:6379 is back under the alert thresholds.",
  "timestamp": 1706457660000,
  "instance": { "host": "10.0.0.1", "port": 6379 },
  "connectionId": "conn-1"
}

cluster.failover (Pro) 

{
  "clusterState": "fail",
  "previousState": "ok",
  "slotsAssigned": 16384,
  "slotsFailed": 128,
  "knownNodes": 6,
  "message": "Cluster state changed from ok to fail",
  "timestamp": 1706457600000
}

anomaly.detected (Pro) 

{
  "anomalyId": "anom-550e8400-e29b-41d4-a716-446655440000",
  "metricType": "ops_per_sec",
  "anomalyType": "spike",
  "severity": "warning",
  "value": 50000,
  "baseline": 10000,
  "stdDev": 5000,
  "zScore": 4.2,
  "threshold": 3.0,
  "message": "Unusual spike in ops_per_sec: 50000 (baseline: 10000, z-score: 4.2)",
  "timestamp": 1706457600000,
  "incidentUrl": "https://monitor.example.com/anomalies/incidents/by-event/anom-550e8400-e29b-41d4-a716-446655440000?at=1706457600000&connection=conn-1"
}

incidentUrl opens the incident that holds the event. It is only present when FRONTEND_URL is set.

A finding on a primary that a cluster connection reads but that is not registered, such as reshard_shard_skew or hot_shard, also carries nodeAddr, and its sourceHost and sourcePort are that primary’s address rather than the registered node’s. A topology_divergence event carries them too when exactly one node disagrees with the rest; when several nodes stand on each side it has no nodeAddr, and the nodes are named in the message (see topology_divergence).

Key names in hot_shard messages. On a cluster with an LFU maxmemory-policy, the message of a hot_shard event names up to 3 of the hottest keys it sampled on the shard, for example Hot keys: "user:42" (freq 255), "cart:{9}" (freq 201). (see hot keys). The sample is the first 500 names of each of the shard’s top three slots, so a hotter key can be missing. If your key names hold personal or otherwise sensitive data, such as an email address or a session token, check where the webhook delivers before you subscribe a chat channel or a third-party tool to anomaly.detected. Each name is cut to 64 characters, and control and invisible characters are written as \xNN or \u{NNNN}, so a name cannot add a line. The Slack and Discord formats also escape mentions, links and markdown. The generic format carries the text as it is in a JSON string, so escape it yourself if you forward it to a chat tool.

anomaly.sequence.detected (Pro) 

{
  "correlationId": "7d0f6f0e-3b0a-4a51-9d57-2f6f6c1b8a10",
  "pattern": "backlog_exhaustion",
  "severity": "critical",
  "diagnosis": "The replication backlog had shrunk below the time a replica needs to reconnect, and a later disconnect outran it and forced a full resync.",
  "recommendations": [
    "Raise repl-backlog-size so a disconnect of this length resumes with a partial sync",
    "Check why the replica disconnected",
    "Act on backlog coverage warnings before a disconnect turns them into a full resync"
  ],
  "values": [
    { "setting": "repl-backlog-size", "current": 1, "target": 64, "unit": "MB" }
  ],
  "anomalyIds": ["anom-1", "anom-2"],
  "metricTypes": ["backlog_coverage_risk", "resync_escalation"],
  "anomalyId": "anom-1",
  "timestamp": 1706457600000,
  "message": "Backlog exhaustion: The replication backlog had shrunk below the time a replica needs to reconnect, and a later disconnect outran it and forced a full resync.",
  "incidentUrl": "https://monitor.example.com/anomalies/incidents/by-event/anom-1?at=1706457600000&connection=conn-1",
  "connectionId": "conn-1"
}

Notes:

  • anomalyId and timestamp are the first event listed in anomalyIds. The alert is sent under that event’s connection. A sequence can span several nodes of a cluster; a webhook scoped only to another connection of the same sequence does not receive it.
  • incidentUrl opens the incident that holds the sequence. It is only present when FRONTEND_URL is set.
  • values is only present when the match carries settings to change.
  • Sent once per recorded match, and again when a compound rule replaces it with a more severe diagnosis. Nothing is sent on recovery.

cve.critical_detected (Pro)

Fired only when a CVE scan finds new critical findings vs the previous stored scan (fire on change — dataset refresh alone does not spam).

Notes:

  • The first scan after enabling CVE (or upgrading) only establishes the baseline and does not fire.
  • Degraded scans still fire with "partial": true (unreachable nodes, missing sources, unknown topology/modules): counts are a floor, not a ceiling, so a permanently degraded scan cannot mute alerting forever. Findings on nodes that first appear after a partial baseline stay suppressed so recovery from a blip does not page. When module inventory is unknown for a node, its last-known module findings are carried forward, so a transient MODULE LIST failure neither hides nor re-alerts them; genuinely new module CVEs still fire on recovery.
  • A single finding that is both critical and KEV-exploited emits both cve.critical_detected (Pro) and cve.kev_detected (Enterprise) plus one OTel event each, so subscribe accordingly.
  • Findings matched only through NVD data for a branch the vendor advisory doesn’t list (typically an end-of-life branch, shown with confidence broad) are shown in the UI but don’t fire these events unless CVE_ALERT_ON_BROAD=true. Other findings fire regardless of confidence.
{
  "criticalCount": 2,
  "kevCount": 1,
  "fingerprint": "a1b2c3d4e5f60718",
  "datasetVersion": "2026-09-15T00:00:00Z",
  "topFindings": [
    { "cveId": "CVE-2026-12345", "severity": "critical", "knownExploited": true, "fixedIn": "8.0.10", "confidence": "exact" }
  ],
  "drift": false,
  "partial": false,
  "message": "New critical CVEs detected (2 critical, 1 exploited (KEV)) on connection conn-1",
  "timestamp": 1706457600000
}

scaling_readiness.low (Pro)

Fires once when a connection’s scaling readiness score drops to or below its alert threshold (default 40, maximum 80) and re-arms after the score recovers above the hysteresis margin. Configure the threshold on the Scaling Readiness page.

{
  "score": 32,
  "threshold": 40,
  "band": "red",
  "bindingDimension": "memory",
  "summary": "Memory is your binding constraint (91% of 4 GB).",
  "dimensions": [
    { "key": "memory", "score": 12, "excludedReason": null },
    { "key": "connections", "score": null, "excludedReason": "Not reported over OTLP" }
  ],
  "message": "Scaling readiness 32 dropped to 40 or below: Memory is your binding constraint (91% of 4 GB).",
  "timestamp": 1735689600000
}

kv_cache.hit_rate_low (Pro)

Fires once per engine and model when the LMCache token hit rate over the last 15 minutes drops to or below the connection’s threshold (default 0.2, maximum 0.9). A window is checked only when it holds at least 10,000 requested tokens. It re-arms after the hit rate recovers above threshold × 1.1 (the fixed global hysteresis factor of 0.9, mirrored for a below-threshold alert), so a 0.2 threshold re-arms above 0.22. A webhook’s own alertConfig.hysteresisFactor does not apply to this event. Configure the threshold on the KV Cache page; see KV Cache Monitoring.

{
  "connectionId": "conn-1",
  "engineId": "9d1f6c52-3a7e-4b1c-8f2d-5e0a4c7b9d13",
  "engineName": "vllm-prod",
  "model": "meta-llama/Llama-3.1-8B-Instruct",
  "hitRate": 0.12,
  "threshold": 0.2,
  "requestedTokens": 184320,
  "windowMs": 900000,
  "message": "LMCache hit rate for meta-llama/Llama-3.1-8B-Instruct on vllm-prod is 12.0% (threshold 20.0%)",
  "timestamp": 1735689600000,
  "instance": { "host": "valkey.example.com", "port": 6379 }
}

kv_cache.eviction_risk (Pro)

Fires once per connection and reason when a footprint collection finds LMCache at risk of losing or being unable to shed keys, and re-arms when the condition clears (active returns to false). unevictable clears on the first collection where it no longer holds; evicting clears only after 3 collections in a row without evictions under memory pressure (about 15 minutes at the default KV_CACHE_FOOTPRINT_INTERVAL_MS). reason is unevictable (the policy starts with volatile- and at least 90% of the sampled LMCache keys have no TTL) or evicting (keys were evicted since the last collection, used_memory is at least 90% of maxmemory, and LMCache holds at least half of used_memory). evictedKeysDelta is null on the first collection after startup. lmcacheMemoryShare and noTtlRatio are fractions between 0 and 1.

{
  "connectionId": "conn-1",
  "reason": "unevictable",
  "active": true,
  "policy": "volatile-lru",
  "usedMemory": 3758096384,
  "maxmemory": 4294967296,
  "lmcacheMemoryShare": 0.92,
  "noTtlRatio": 1,
  "evictedKeysDelta": null,
  "message": "LMCache keys have no TTL under volatile-lru; Valkey cannot evict them",
  "timestamp": 1735689600000,
  "instance": { "host": "valkey.example.com", "port": 6379 }
}

retry_amplification.detected (Pro)

Fires when the anomaly detector sees a self-sustaining retry loop rather than a transient blip. Polls are grouped into 60-second windows and compared with the median of the last 30 normal windows (detection starts after 10). A window is hot when commands per client are at least 1.3× baseline, ops/s at least 1.2× baseline and the client count within ±15% of baseline, while a P99 latency regression, an error-reply burst or connection churn is present. A burst shorter than 3 hot windows is treated as transient and never alerts. After 3 hot windows the event fires in one of two shapes, given in pattern:

  • rising: load rises at least 10% and moves with stress inside the run (correlation ≥ 0.5).
  • step: the last 3 windows hold at 2× baseline or more with an error-reply or reconnect burst. correlation is null for this shape.

level is critical while saturation is active in the latest window, otherwise warning. It fires when a run first reaches a level and each time it rises to critical again after falling back to warning. A fall to warning does not fire. It is sent when the event is first saved, so it waits for storage if a save has to be retried, and a finding that is never saved sends nothing. It re-arms after 2 normal windows.

This event replaces the generic anomaly.detected for retry amplification, so one finding is delivered once.

trigger is the stress signal that showed up first: latency, errors or churn. loadRatio is commands per client against baseline; value is the same ratio rounded to two decimals against a baseline of 1, which is what Slack and Discord show. topContributors lists up to 5 clients grouped by name, then user, then IP; a connection captured before it ran CLIENT SETNAME counts under the name it set later, and BetterDB’s own connections are left out. commandRateGrowth compares each client’s command rate during the episode with its rate just before the episode and needs CLIENT LIST tot-cmds (Valkey 8+). It is null for a client with no reading from before the episode. On older servers it is null for every client, and clients are ranked by newConnections (reconnects seen in client snapshots). topContributors is null when neither source could be read.

On a cluster or Sentinel connection the finding can come from another primary the connection reads rather than the registered node. instance is then that primary’s address, connectionId stays the cluster or Sentinel connection, and topContributors is null.

{
  "metricType": "retry_amplification",
  "value": 1.9,
  "baseline": 1,
  "level": "warning",
  "trigger": "latency",
  "windows": 3,
  "loadRatio": 1.9,
  "pattern": "rising",
  "correlation": 0.97,
  "topContributors": [
    { "key": "checkout", "keyKind": "name", "commandRateGrowth": 4.8, "newConnections": 12, "connections": 40 }
  ],
  "message": "WARNING: Retry amplification — commands per client at 1.9× baseline for 3 min while connected clients stay flat at ~120; load is rising with stress (r=0.97). Trigger: a latency regression (COMMAND_P99). Without client backoff the loop keeps feeding itself. Top clients: name checkout (cmds 4.8×, 12 reconnects).",
  "timestamp": 1735689600000,
  "instance": { "host": "valkey.example.com", "port": 6379 }
}

cve.kev_detected (Enterprise)

Fired only when a CVE scan finds new CISA KEV-exploited findings vs the previous stored scan, even when severity is below critical.

{
  "kevCount": 1,
  "criticalCount": 0,
  "fingerprint": "a1b2c3d4e5f60718",
  "datasetVersion": "2026-09-15T00:00:00Z",
  "topFindings": [
    { "cveId": "CVE-2026-54321", "severity": "high", "knownExploited": true, "fixedIn": "8.0.10", "confidence": "exact" }
  ],
  "drift": false,
  "partial": false,
  "message": "New exploited (KEV) CVEs detected (1 KEV) on connection conn-1",
  "timestamp": 1706457600000
}

compliance.alert (Enterprise) 

{
  "complianceType": "data_retention",
  "severity": "high",
  "memoryUsedPercent": 92.5,
  "maxmemoryPolicy": "noeviction",
  "message": "Compliance alert: Memory at 92.5% with noeviction policy - data loss risk",
  "timestamp": 1706457600000
}

audit.policy.violation (Enterprise) 

{
  "username": "readonly_user",
  "clientInfo": "10.0.1.50:43210",
  "violationType": "command",
  "violatedCommand": "SET",
  "count": 3,
  "message": "ACL command violation by readonly_user@10.0.1.50:43210 (count: 3)",
  "timestamp": 1706457600000
}

acl.violation (Enterprise) 

{
  "username": "app_user",
  "command": "FLUSHALL",
  "reason": "command",
  "message": "ACL access denied: app_user attempted FLUSHALL",
  "timestamp": 1706457600000
}

acl.modified (Enterprise) 

{
  "modifiedBy": "admin",
  "changeType": "user_added",
  "affectedUser": "new_user",
  "message": "ACL configuration changed: user_added (user: new_user)",
  "timestamp": 1706457600000
}

config.changed (Enterprise) 

{
  "configKey": "maxmemory-policy",
  "oldValue": "noeviction",
  "newValue": "allkeys-lru",
  "modifiedBy": "admin",
  "message": "Configuration changed: maxmemory-policy = allkeys-lru (was: noeviction)",
  "timestamp": 1706457600000
}

Signature Verification

If you configure a webhook secret, BetterDB signs each request using HMAC-SHA256. This allows you to verify that requests are genuinely from your BetterDB instance.

Signature Algorithm 

signature = HMAC-SHA256(secret, timestamp + "." + payload)

The signature is sent in the X-Webhook-Signature header and the timestamp in X-Webhook-Timestamp.

Verification (Node.js) 

const crypto = require('crypto');

function verifyWebhook(req) {
  const signature = req.headers['x-webhook-signature'];
  const timestamp = req.headers['x-webhook-timestamp'];
  const payload = JSON.stringify(req.body);
  const secret = process.env.WEBHOOK_SECRET;

  // Construct signed payload
  const signedPayload = `${timestamp}.${payload}`;

  // Calculate expected signature
  const expectedSig = crypto
    .createHmac('sha256', secret)
    .update(signedPayload)
    .digest('hex');

  // Timing-safe comparison
  return crypto.timingSafeEqual(
    Buffer.from(signature),
    Buffer.from(expectedSig)
  );
}

// Express.js example
app.post('/webhooks/betterdb', express.raw({ type: 'application/json' }), (req, res) => {
  if (!verifyWebhook(req)) {
    return res.status(401).send('Invalid signature');
  }

  const event = JSON.parse(req.body);
  console.log('Received event:', event.event);

  // Process webhook...

  res.status(200).send('OK');
});

Verification (Python) 

import hmac
import hashlib
from flask import Flask, request, abort

app = Flask(__name__)
WEBHOOK_SECRET = 'your-secret-key'

def verify_webhook(request):
    signature = request.headers.get('X-Webhook-Signature')
    timestamp = request.headers.get('X-Webhook-Timestamp')
    payload = request.get_data(as_text=True)

    # Construct signed payload
    signed_payload = f"{timestamp}.{payload}"

    # Calculate expected signature
    expected = hmac.new(
        WEBHOOK_SECRET.encode(),
        signed_payload.encode(),
        hashlib.sha256
    ).hexdigest()

    # Timing-safe comparison
    return hmac.compare_digest(signature, expected)

@app.route('/webhooks/betterdb', methods=['POST'])
def handle_webhook():
    if not verify_webhook(request):
        abort(401, 'Invalid signature')

    event = request.get_json()
    print(f"Received event: {event['event']}")

    # Process webhook...

    return 'OK', 200

Verification (Go) 

package main

import (
    "crypto/hmac"
    "crypto/sha256"
    "crypto/subtle"
    "encoding/hex"
    "fmt"
    "io"
    "net/http"
)

const webhookSecret = "your-secret-key"

func verifyWebhook(r *http.Request, body []byte) bool {
    signature := r.Header.Get("X-Webhook-Signature")
    timestamp := r.Header.Get("X-Webhook-Timestamp")

    // Construct signed payload
    signedPayload := timestamp + "." + string(body)

    // Calculate expected signature
    h := hmac.New(sha256.New, []byte(webhookSecret))
    h.Write([]byte(signedPayload))
    expected := hex.EncodeToString(h.Sum(nil))

    // Timing-safe comparison
    return subtle.ConstantTimeCompare([]byte(signature), []byte(expected)) == 1
}

func handleWebhook(w http.ResponseWriter, r *http.Request) {
    body, _ := io.ReadAll(r.Body)

    if !verifyWebhook(r, body) {
        http.Error(w, "Invalid signature", http.StatusUnauthorized)
        return
    }

    fmt.Println("Webhook verified!")
    w.WriteHeader(http.StatusOK)
}

Replay Attack Prevention

The timestamp in X-Webhook-Timestamp should be validated to prevent replay attacks. Reject requests with timestamps older than 5 minutes:

function isRecentTimestamp(timestamp) {
  const now = Date.now();
  const age = now - parseInt(timestamp);
  return age < 5 * 60 * 1000; // 5 minutes
}

Retry Policy

Failed webhook deliveries are automatically retried with exponential backoff. This ensures transient network issues don’t cause missed notifications.

Default Retry Behavior

Attempt Delay Total Elapsed
1 Immediate 0s
2 1 second 1s
3 2 seconds 3s
4 4 seconds 7s

Default Configuration:

  • Max retries: 3
  • Initial delay: 1000ms
  • Backoff multiplier: 2x
  • Max delay: 60000ms (1 minute)

Custom Retry Policy

You can configure custom retry policies per webhook:

{
  "name": "Critical Alerts",
  "url": "https://api.example.com/webhooks/critical",
  "events": ["instance.down"],
  "retryPolicy": {
    "maxRetries": 5,
    "initialDelayMs": 2000,
    "backoffMultiplier": 2,
    "maxDelayMs": 300000
  }
}

Dead Letter Queue

After all retries are exhausted, the delivery moves to dead letter status. Dead letter deliveries can be:

  • Viewed in the webhook delivery history (UI or API)
  • Manually retried via the UI or API
  • Used for alerting on persistent failures
# View webhook deliveries (including dead letters)
curl -b cookies.txt -c cookies.txt http://localhost:3001/api/webhooks/{id}/deliveries

# Manually retry a dead letter delivery
curl -b cookies.txt -c cookies.txt -X POST http://localhost:3001/api/webhooks/deliveries/{deliveryId}/retry

Storage Failures Before Delivery

Every delivery is written to storage before the HTTP request is sent, and the retry processor works from those rows. If storage is unavailable when an event fires (the subscribed-webhook lookup or the delivery write fails), the event is held in an in-memory buffer instead of being dropped:

  • The buffer is retried with exponential backoff: 1 second, doubling on each failed attempt, capped at 1 minute.
  • Once storage recovers, buffered events are dispatched in the order they fired, with their original timestamps. Events raised while the buffer is non-empty are queued behind it, and a retry waits for any dispatch that was already in flight when the outage began, so for example an instance.up is never delivered before the instance.down that preceded it.
  • If one webhook’s delivery row was written and another’s was not, only the missing one is retried. If a write failed ambiguously (the row may have been stored), the retry checks the webhook’s 100 most recent deliveries for the same payload ID before writing again. If more than 100 newer deliveries for that webhook were written in the meantime (heavy concurrent traffic), an ambiguously written row can fall outside that window and the event can be delivered twice.
  • The buffer holds at most 1,000 events. When it is full the oldest event is dropped and an error is logged.
  • The buffer lives in process memory. Events still buffered when the process stops or restarts are lost (a warning with the count is logged on shutdown).

GET /api/webhooks/stats/retry-queue reports bufferedEvents (currently held in the buffer) and droppedEvents (dropped because the buffer was full, since the process started). The same endpoint reads storage, so it may fail while storage is down; the dispatcher logs each buffered and dropped event.

Success Criteria

A webhook delivery is considered successful when:

  • HTTP response status is 2xx (200-299)
  • Response received within timeout (default: 30 seconds)

Any other response (4xx, 5xx, timeout, network error) triggers a retry.

Per-Webhook Configuration

Each webhook can be individually configured with custom delivery settings, alert behavior, and thresholds. This enables different notification channels to have different sensitivity levels.

Delivery Configuration

Control how webhook requests are sent:

{
  "name": "Slow Endpoint",
  "url": "https://slow-api.example.com/webhook",
  "events": ["instance.down"],
  "deliveryConfig": {
    "timeoutMs": 60000,
    "maxResponseBodyBytes": 50000
  }
}
Field Type Default Description
timeoutMs integer 30000 Request timeout in milliseconds (1000-120000)
maxResponseBodyBytes integer 10000 Max response body to store (1000-100000)

Alert Configuration

Control hysteresis behavior for threshold-based alerts:

{
  "name": "Sensitive Alerts",
  "url": "https://api.example.com/webhook",
  "events": ["memory.critical"],
  "alertConfig": {
    "hysteresisFactor": 0.95
  }
}
Field Type Default Description
hysteresisFactor number 0.9 Recovery threshold = trigger x factor (0.5-0.99)

How hysteresis works:

  • Alert fires when metric crosses threshold
  • Alert clears when metric drops below threshold x hysteresisFactor
  • Example: 90% threshold with 0.9 factor -> clears at 81%
  • Lower factor = wider buffer = fewer false re-triggers

Custom Thresholds

Override default alert thresholds per webhook. This enables different notification channels for different severity levels:

{
  "name": "Early Warning - Slack",
  "url": "https://hooks.slack.com/services/...",
  "payloadFormat": "slack",
  "events": ["memory.critical", "connection.critical"],
  "thresholds": {
    "memoryCriticalPercent": 75,
    "connectionCriticalPercent": 70
  }
}

{
  "name": "Critical - PagerDuty",
  "url": "https://events.pagerduty.com/...",
  "events": ["memory.critical", "connection.critical"],
  "thresholds": {
    "memoryCriticalPercent": 95,
    "connectionCriticalPercent": 95
  }
}
Field Type Default Applies To
memoryCriticalPercent integer 90 memory.critical
connectionCriticalPercent integer 90 connection.critical
complianceMemoryPercent integer 80 compliance.alert
slowlogCount integer 100 slowlog.threshold
replicationLagSeconds integer 10 replication.lag
latencySpikeMs integer 0 (auto) latency.spike
connectionSpikeCount integer 0 (auto) connection.spike

Notes:

  • Thresholds only apply to events the webhook subscribes to
  • Value of 0 for spike thresholds means “use automatic baseline detection”
  • Each webhook tracks alert state independently (one webhook recovering doesn’t affect others)

Full Configuration Example 

{
  "name": "Production Critical Alerts",
  "url": "https://api.pagerduty.com/webhooks",
  "secret": "whsec_abc123",
  "enabled": true,
  "events": ["instance.down", "instance.up", "memory.critical", "connection.critical"],
  "headers": {
    "X-Routing-Key": "prod-database-alerts"
  },
  "retryPolicy": {
    "maxRetries": 5,
    "initialDelayMs": 2000,
    "backoffMultiplier": 2,
    "maxDelayMs": 300000
  },
  "deliveryConfig": {
    "timeoutMs": 10000,
    "maxResponseBodyBytes": 5000
  },
  "alertConfig": {
    "hysteresisFactor": 0.85
  },
  "thresholds": {
    "memoryCriticalPercent": 95,
    "connectionCriticalPercent": 90
  }
}

Rate Limiting & Hysteresis

To prevent alert fatigue from metrics oscillating around thresholds, BetterDB implements hysteresis:

How Hysteresis Works

  • Alert fires when metric exceeds threshold
  • Alert clears only when metric drops below 90% of threshold
  • This creates a 10% “buffer zone” to prevent rapid on/off alerts

Example: Memory alert at 90% threshold

  • Alert fires when memory reaches 90%
  • Alert clears when memory drops below 81% (90% × 0.9)
  • If memory oscillates between 89% and 91%, alert fires once and stays active

Alert State Tracking

BetterDB maintains alert state in memory using an LRU cache:

  • Max alerts: 10,000 by default, one entry per alert type × connection × webhook (plus one OTLP compliance entry per connection). Raise it for large fleets with the WEBHOOK_ALERT_STATE_MAX environment variable.
  • TTL: 24 hours. A condition that is still breached after a day re-fires once as a reminder.
  • Eviction: Oldest alerts evicted if limit exceeded; an evicted alert that is still breached re-fires on the next poll

Threshold-Based Events

The following events use configurable thresholds with hysteresis:

Event Default Threshold Config Key
memory.critical 90% memoryCriticalPercent
connection.critical 90% connectionCriticalPercent
slowlog.threshold (Pro) 100 entries slowlogCount
replication.lag (Pro) 10 seconds replicationLagSeconds
latency.spike (Pro) 0 (auto) latencySpikeMs
connection.spike (Pro) 0 (auto) connectionSpikeCount
compliance.alert (Enterprise) 80% complianceMemoryPercent

See Custom Thresholds for per-webhook configuration.

Non-Threshold Events

These events fire immediately without hysteresis:

  • instance.down / instance.up (state change events)
  • client.blocked (each occurrence)
  • anomaly.detected (each detection, except command_p99, retry_amplification and replication_divergence, which have their own events)
  • anomaly.sequence.detected (each recorded sequence match)
  • acl.violation (each violation)
  • audit.policy.violation (each violation)
  • acl.modified (each change)
  • config.changed (each change)

writes.rejected fires immediately; writes.recovered waits for a 60s quiet window (see writes.rejected / writes.recovered above).

API Reference

List Webhooks 

GET /api/webhooks

Returns all webhooks for the current instance with secrets redacted.

Response:

[
  {
    "id": "550e8400-e29b-41d4-a716-446655440000",
    "name": "Production Alerts",
    "url": "https://api.example.com/webhooks",
    "secret": "whsec_abc1***",
    "enabled": true,
    "events": ["instance.down", "instance.up"],
    "headers": {},
    "retryPolicy": {
      "maxRetries": 3,
      "backoffMultiplier": 2,
      "initialDelayMs": 1000,
      "maxDelayMs": 60000
    },
    "createdAt": 1706457600000,
    "updatedAt": 1706457600000
  }
]

Create Webhook 

POST /api/webhooks
Content-Type: application/json

{
  "name": "Production Alerts",
  "url": "https://api.example.com/webhooks/betterdb",
  "secret": "your-secret-key",
  "events": ["instance.down", "instance.up", "memory.critical"],
  "enabled": true,
  "headers": {
    "X-Custom-Header": "value"
  },
  "retryPolicy": {
    "maxRetries": 5,
    "initialDelayMs": 2000,
    "backoffMultiplier": 2,
    "maxDelayMs": 300000
  }
}

Response: 201 Created with webhook object (secret redacted)

Validation:

  • URL must be valid HTTPS (HTTP allowed in development)
  • Events must be valid webhook event types
  • Events must match your license tier (403 if attempting to subscribe to locked events)
  • Custom headers cannot include restricted headers (Host, Content-Length, etc.)

Update Webhook 

PUT /api/webhooks/{id}
Content-Type: application/json

{
  "name": "Updated Name",
  "enabled": false,
  "events": ["instance.down"]
}

Response: 200 OK with updated webhook object

Delete Webhook 

DELETE /api/webhooks/{id}

Response: 204 No Content

Note: Deleting a webhook also deletes all associated delivery history.

Test Webhook 

POST /api/webhooks/{id}/test

Sends a test payload to verify connectivity. Uses the first subscribed event type or instance.down as fallback.

Response:

{
  "success": true,
  "statusCode": 200,
  "responseBody": "OK",
  "durationMs": 145
}

Get Webhook Deliveries 

GET /api/webhooks/{id}/deliveries?limit=50&offset=0

Response:

[
  {
    "id": "del-550e8400-e29b-41d4-a716-446655440000",
    "webhookId": "550e8400-e29b-41d4-a716-446655440000",
    "eventType": "instance.down",
    "status": "success",
    "statusCode": 200,
    "responseBody": "OK",
    "attempts": 1,
    "durationMs": 145,
    "createdAt": 1706457600000,
    "completedAt": 1706457600145
  }
]

Delivery Status Values:

  • pending - Queued for delivery
  • success - Delivered successfully (2xx response)
  • retrying - Failed, will retry
  • dead_letter - All retries exhausted
  • failed - Permanent failure (not retryable)

Retry Failed Delivery 

POST /api/webhooks/deliveries/{deliveryId}/retry

Manually retry a failed or dead letter delivery.

Response: 202 Accepted with message “Retry queued”

Get Allowed Events 

GET /api/webhooks/allowed-events

Returns webhook events allowed for your current license tier.

Response:

{
  "tier": "pro",
  "allowedEvents": [
    "instance.down",
    "instance.up",
    "memory.critical",
    "connection.critical",
    "client.blocked",
    "writes.rejected",
    "writes.recovered",
    "slowlog.threshold",
    "replication.lag",
    "cluster.failover",
    "anomaly.detected",
    "latency.spike",
    "connection.spike"
  ],
  "lockedEvents": [
    "compliance.alert",
    "audit.policy.violation",
    "acl.violation",
    "acl.modified",
    "config.changed"
  ]
}

Get Retry Queue Stats 

GET /api/webhooks/stats/retry-queue

Response:

{
  "pendingRetries": 5,
  "nextRetryTime": 1706457665000,
  "bufferedEvents": 0,
  "droppedEvents": 0
}

Receivers on a Private Network

By default BetterDB only delivers webhooks to public addresses. In production (NODE_ENV=production, which the Docker image and the Helm chart set) a webhook URL is rejected when it points at a private, loopback or link-local address, or at a hostname that resolves to one. This protects against server-side request forgery (SSRF): without it, anyone who can create a webhook could make BetterDB call services on its own network.

A self-hosted BetterDB often needs to reach a receiver on that network: an alert relay in the same Kubernetes cluster, an n8n instance, a self-hosted chat server. Two environment variables allow specific private targets. Both are empty by default.

Variable Meaning
WEBHOOK_ALLOWED_PRIVATE_CIDRS Comma-separated IPv4/IPv6 CIDRs, e.g. 10.244.0.0/16,10.96.0.0/12. A webhook address inside one of them is accepted even though it is private.
WEBHOOK_ALLOWED_PRIVATE_HOSTS Comma-separated hostnames, exact (relay.internal) or suffix wildcards (*.svc.cluster.local). A URL whose hostname matches may resolve to a private address.

The same allowlist applies when a webhook is created, updated, tested and on every delivery and retry, so a webhook that saves also delivers.

Examples

Kubernetes, by Service DNS name (the Helm chart values render the same variables):

# values.yaml
webhooks:
  allowedPrivateHosts: ["*.svc.cluster.local"]

Kubernetes, by pod and Service CIDR:

# values.yaml
webhooks:
  allowedPrivateCidrs: ["10.244.0.0/16", "10.96.0.0/12"]

Docker, a receiver on the default bridge network:

docker run -e WEBHOOK_ALLOWED_PRIVATE_CIDRS=172.17.0.0/16 ... betterdb/monitor

Rules

  • Metadata and link-local addresses stay blocked. 169.254.0.0/16, fe80::/10, fd00:ec2::254, 100.100.100.200 and 192.0.0.192 can never be allowed. A CIDR that overlaps one of them (for example 0.0.0.0/0, 169.254.0.0/16, or the whole 100.64.0.0/10 or fd00::/8 range) is refused at startup; list narrower ranges instead.
  • Loopback needs its own CIDR. 127.0.0.0/8, ::1 and 0.0.0.0/8 are accepted only when listed as a CIDR, and never through a hostname pattern, so a webhook cannot reach BetterDB’s own API or a sidecar by accident. Use the IP literal (http://127.0.0.1:9000/hook); the name localhost stays rejected.
  • The allowlist applies in every environment. Without one, localhost and 127.x.x.x are allowed when NODE_ENV is not production. Once either variable is set, that exception no longer applies and loopback must be listed like any other private range.
  • A typo fails startup. An invalid CIDR or hostname pattern stops BetterDB from starting with an error naming the entry. * alone is not a valid hostname pattern.
  • Ignored in cloud mode. With CLOUD_MODE set, both variables are ignored and a warning is logged at startup.
  • Shrinking the allowlist blocks existing webhooks. Deliveries to a target that is no longer allowed fail with Webhook URL resolves to a blocked address and are recorded as failed deliveries.

Security Considerations

  • Anyone who can create a webhook can make BetterDB send a POST request to anything in the allowlist. List the narrowest CIDR or hostname that works.
  • Allow a hostname only when you control its DNS, such as cluster DNS or a private zone. Whoever controls the DNS for an allowed name decides which private address it reaches. BetterDB checks the address it actually connects to and does not follow redirects, so a name that changes its answer between the check and the request (DNS rebinding) is still held to the allowlist.

Troubleshooting

Webhook Not Firing

Check webhook configuration:

  1. Verify webhook is enabled (enabled: true)
  2. Confirm you’re subscribed to the event type
  3. Check your license tier includes the event: 
    curl -b cookies.txt -c cookies.txt http://localhost:3001/api/webhooks/allowed-events
  4. Review delivery history for errors: 
    curl -b cookies.txt -c cookies.txt http://localhost:3001/api/webhooks/{id}/deliveries

Check event is being triggered:

  • For memory.critical, verify memory is actually exceeding 90%
  • For anomaly.detected, ensure anomaly detection is enabled (Pro tier required)
  • For threshold events, remember hysteresis: alert fires once and won’t re-fire until recovery

Verify endpoint reachability:

# Test webhook connectivity
curl -b cookies.txt -c cookies.txt -X POST http://localhost:3001/api/webhooks/{id}/test

Signature Verification Failing

Common issues:

  1. Using parsed JSON instead of raw body
    • Solution: Use raw body string for signature verification
    • Express.js: express.raw({ type: 'application/json' })
    • Flask: request.get_data(as_text=True)
  2. Timestamp not included in signed payload
    • Correct format: timestamp + "." + payload
    • Example: 1706457600000.{"event":"instance.down",...}
  3. Secret mismatch
    • Verify secret matches exactly (no trailing whitespace)
    • Check secret wasn’t redacted when retrieving webhook details
  4. Character encoding issues
    • Ensure UTF-8 encoding for all strings
    • Use Buffer.from(str, 'utf8') in Node.js

Debug signature generation:

console.log('Timestamp:', req.headers['x-webhook-timestamp']);
console.log('Payload:', req.body.toString());
console.log('Signed payload:', `${timestamp}.${payload}`);
console.log('Expected signature:', expectedSig);
console.log('Received signature:', signature);

Too Many Alerts

Threshold oscillation:

  • Hysteresis prevents rapid on/off, but sustained oscillation will still alert
  • Solution: Adjust thresholds to appropriate levels for your workload
  • For advanced alerting logic, consider using Prometheus + Alertmanager

Alert not clearing:

  • Alerts use 10% hysteresis - metric must drop significantly below threshold
  • Example: 90% memory alert clears at 81%, not 89%
  • Check if metric is genuinely recovering

Multiple webhook subscriptions:

  • If multiple webhooks subscribe to the same event, all will fire
  • Review webhook configuration to avoid duplicates

High Delivery Latency

Network issues:

  • Check network connectivity between BetterDB and your endpoint
  • Consider using a webhook relay service for reliability
  • Verify your endpoint responds quickly (< 5 seconds recommended)

Timeout configuration:

  • Default timeout: 30 seconds
  • Long-running endpoints may need optimization
  • Consider async processing: return 200 immediately, process in background

Retry backlog:

  • Check retry queue stats: GET /api/webhooks/stats/retry-queue
  • High pending retries indicate delivery issues
  • Review dead letter queue for patterns

SSRF Security Error

BetterDB blocks private IP addresses to prevent SSRF attacks:

Blocked in production:

  • localhost, 127.0.0.1, 0.0.0.0
  • Private IP ranges: 10.x.x.x, 172.16-31.x.x, 192.168.x.x
  • Link-local: 169.254.x.x
  • IPv6 private ranges

Allowed in development:

  • localhost and 127.x.x.x are allowed when NODE_ENV !== 'production' and no private-network allowlist is set

Solution:

  • For a receiver on your own network (same Kubernetes cluster, VPC or Docker network), allow it with WEBHOOK_ALLOWED_PRIVATE_CIDRS or WEBHOOK_ALLOWED_PRIVATE_HOSTS; see Receivers on a Private Network
  • Otherwise use public endpoints or a reverse proxy
  • For testing, set NODE_ENV=development

Delivery History Missing

Memory storage:

  • STORAGE_TYPE=memory loses delivery history on restart
  • Consider PostgreSQL for persistent storage

Retention limits:

  • Memory adapter keeps last 1000 deliveries per webhook
  • PostgreSQL has no artificial limits

Pruning:

  • Old deliveries can be manually pruned via API
  • Configure retention policies as needed

Integration Examples

Slack 

curl -b cookies.txt -c cookies.txt -X POST http://localhost:3001/api/webhooks \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Slack Notifications",
    "url": "https://hooks.slack.com/services/YOUR/SLACK/WEBHOOK",
    "events": ["instance.down", "memory.critical"],
    "enabled": true
  }'

Your endpoint should transform BetterDB payloads to Slack’s format.

PagerDuty 

curl -b cookies.txt -c cookies.txt -X POST http://localhost:3001/api/webhooks \
  -H "Content-Type: application/json" \
  -d '{
    "name": "PagerDuty Incidents",
    "url": "https://events.pagerduty.com/v2/enqueue",
    "headers": {
      "Authorization": "Token token=YOUR_INTEGRATION_KEY"
    },
    "events": ["instance.down", "cluster.failover"],
    "enabled": true
  }'

Discord 

curl -b cookies.txt -c cookies.txt -X POST http://localhost:3001/api/webhooks \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Discord Alerts",
    "url": "https://discord.com/api/webhooks/YOUR/DISCORD/WEBHOOK",
    "events": ["instance.down", "instance.up"],
    "enabled": true
  }'

Microsoft Teams 

curl -b cookies.txt -c cookies.txt -X POST http://localhost:3001/api/webhooks \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Teams Notifications",
    "url": "https://outlook.office.com/webhook/YOUR/TEAMS/WEBHOOK",
    "events": ["memory.critical", "connection.critical"],
    "enabled": true
  }'

Custom HTTP Endpoint 

const express = require('express');
const app = express();

app.post('/webhooks/betterdb', express.json(), (req, res) => {
  const { event, timestamp, instance, data } = req.body;

  console.log(`[${new Date(timestamp).toISOString()}] ${event} on ${instance.host}:${instance.port}`);

  // Custom logic based on event type
  switch (event) {
    case 'instance.down':
      // Send SMS alert
      // Create incident ticket
      break;
    case 'memory.critical':
      // Scale up resources
      // Send email to ops team
      break;
    case 'anomaly.detected':
      // Log to anomaly dashboard
      // Trigger auto-remediation
      break;
  }

  res.status(200).send('OK');
});

app.listen(8080);

Best Practices

Security

  1. Always use secrets for production webhooks
  2. Validate signatures on your endpoint
  3. Check timestamps to prevent replay attacks
  4. Use HTTPS for webhook URLs (required in production)
  5. Rotate secrets periodically

Reliability

  1. Return 200 quickly - don’t block webhook handler
  2. Process async - queue work for background processing
  3. Implement retries on your endpoint for transient failures
  4. Monitor delivery history for patterns
  5. Set up dead letter alerts for persistent failures

Performance

  1. Keep response times < 5 seconds to avoid timeouts
  2. Use connection pooling for database operations
  3. Limit payload processing - only extract needed fields
  4. Cache frequently accessed data to reduce latency
  5. Consider webhook relay services for high-volume workloads

Operational

  1. Test webhooks before enabling in production
  2. Monitor delivery success rate via API
  3. Set appropriate retry policies for your SLA
  4. Document your event handling logic
  5. Use multiple webhooks for different alert severities

Further Reading