Skip to content

Architecture

┌──────────────────────────────── Discord ────────────────────────────────┐
│ Gateway (WebSocket) REST API Webhook events │
└───────┬──────────────────────────▲─────────────────────────────┬────────┘
│ events │ calls │ HTTPS, Ed25519-signed
┌───────▼──────────────────────────┴───────────┐ │
│ Your bot process │ │
│ discord.js Client ──► Shardlens SDK │ │
│ • client + REST hooks│ │
│ • HMAC user hashing │ │
│ • bounded queue, gzip│── hashed ──┐ │
└──────────────────────────────────────────────┘ batches ▼ ▼
┌──────────── Collector (Docker or embedded) ────────┐
│ Ingest API · Webhook receiver · Background jobs │
│ Query + live API · Dashboard (React) │
└──────────────────────────┬─────────────────────────┘
│ SQL
┌──────▼──────┐
│ PostgreSQL │
└─────────────┘

The SDK listens to public discord.js events and REST hooks. A listener copies the fields it needs into a plain object and pushes it onto a bounded queue: no I/O and no waiting on the hot path. The user ID is hashed as the event is recorded, with the salt of the period the event falls in (HMAC-SHA-256), so in privacy mode a raw ID never waits in the queue or leaves the process. Every 5 seconds (or sooner when the queue fills) the SDK:

  1. Packs up to 500 events into a batch, gzips it with CompressionStream and sends it with the ingest key.
  2. Keeps the batch on failure and retries with jittered backoff (1 to 60 seconds).

When the queue is full, the least important events are dropped first: gauges, then REST windows, then custom events, and interactions and server changes last. Every drop is counted and reported in SDK health.

  • Ingest checks the key, decompresses with a size cap (1 MB compressed, 5 MB decompressed), validates each event with zod and stores it. Retried batches are recognised by event ID and stored once.
  • Webhook receiver verifies Discord’s Ed25519 signature, answers 204 immediately and processes afterwards.
  • Background jobs run on one collector at a time (an advisory lock elects the leader): minute, hour and day rollups every minute, unique users per salt period every 5 minutes, salt creation and destruction, partition creation and retention, alerts every minute and scheduled reports.
  • Query API answers typed queries. Each query kind maps to a fixed, parameterised SQL template; there is no free-form SQL and no grouping by user.
  • Live API streams live tiles over Server-Sent Events, fed by LISTEN/NOTIFY so it works across collectors.
  • Dashboard is a static React app served by the collector with a strict Content Security Policy.

Raw events go into daily partitions that are dropped whole when they expire (after 35 days by default, 7 to 90 in Settings). Charts read from rollups, all in UTC buckets. Every minute the job leader recomputes the last few minutes, and the minutes late events landed in, then the hours and days they belong to, so every chart is about a minute behind:

What Source Kept Freshness
Live tiles Memory 15 minutes 1–5 seconds
Minute charts Minute rollup 14 days About 1 minute
Charts of ranges up to 13 months Hour rollup 13 months About 1 minute
Older ranges Day rollup For good About 1 minute
Unique users Counts of hashed users Counts for good, hashes until their salt is destroyed About 5 minutes
Command and server details, reliability Raw rows Raw retention Immediate
Mode What runs Status
Standard Caddy → collector (API and dashboard) → PostgreSQL Available
Scaled Several collectors behind a load balancer, one PostgreSQL Available
Embedded Collector inside the bot process on in-process PostgreSQL (PGlite) Available
Exporter only SDK with a Prometheus or OpenTelemetry exporter, no collector Available

The decisions behind this design are recorded as architecture decision records.