Skip to content

Your own database

Your bot may already have a database. Point Shardlens at it, and every event the collector accepts is also written there, so you can query your bot’s usage with the tools you already have, join it with your own data, or keep it for longer.

The dashboard still runs on the collector’s own store: PostgreSQL with Docker, or the built-in database in embedded mode, which needs no database server at all. Its rollups, histograms, unique-user counts and funnels are built on PostgreSQL, so your database holds a copy of the events rather than replacing it.

Set SHARDLENS_SINK_URL on the collector (in Docker Compose, in .env) and restart it:

Terminal window
SHARDLENS_SINK_URL=mongodb+srv://user:password@cluster0.example.mongodb.net/mybot

MongoDB 5 or newer, including Atlas. The database is the one in the URL (mybot here), or shardlens if the URL names none.

In embedded mode, pass the same URL (or a sink of your own) to startEmbeddedCollector:

bot.ts
import { type EventSink, type StoredEvent, startEmbeddedCollector } from '@shardlens/collector/embedded';
// Embedded mode, with every event also copied into the MongoDB the bot already uses. A `mysql://`, `mariadb://`,
// `postgres://` or `sqlite:./events.db` URL works the same way.
const collector = await startEmbeddedCollector({
dataDir: './shardlens-data',
appId: process.env.DISCORD_APP_ID!,
sink: { to: process.env.MONGODB_URI!, retentionDays: 90 },
});
// Or send copies anywhere, by writing a sink yourself. Writes must ignore an event ID they already hold.
declare const myStore: {
upsertMany(rows: StoredEvent[]): Promise<void>;
deleteWhere(filter: { appId: string; userHashes: readonly string[] } | { before: Date }): Promise<number>;
};
const custom: EventSink = {
name: 'my-store',
write: (events) => myStore.upsertMany([...events]),
erase: (appId, userHashes) => myStore.deleteWhere({ appId, userHashes }),
expire: (before) => myStore.deleteWhere({ before }),
close: async () => undefined,
};
const other = await startEmbeddedCollector({
dataDir: './shardlens-other',
appId: process.env.DISCORD_APP_ID!,
port: 8081,
sink: { to: custom, events: ['interaction', 'message_command', 'custom'] },
});
export { collector, other };

The collector log says sink with the database’s name when it starts. If the database is unreachable, the collector starts anyway, logs why once a minute (with any password removed), and keeps trying.

Setting Default What it does
SHARDLENS_SINK_URL off Where to copy events.
SHARDLENS_SINK_TABLE shardlens_events Table or collection name.
SHARDLENS_SINK_RETENTION_DAYS 35 Copies older than this are deleted every six hours (1 to 3,650).
SHARDLENS_SINK_EVENTS every type Comma-separated event types to copy, for example interaction,message_command,custom.

One row (or document) per event, with these columns. In MongoDB the event’s own fields sit at the top level of the document instead of in data, and id is _id.

Column What it holds
id The event’s UUIDv7. A retried batch is stored once.
ts When the event happened (UTC).
app_id Your application ID.
env The SDK’s environment, such as production.
bot_release The SDK’s release, if you set it.
type interaction, message_command, custom, guild_join, rest_window and so on.
guild_id The server, when the event has one.
user_hash The salted user hash (32 hex characters), when the event has a user.
salt_period The salt period of user_hash, such as 2026-11.
data The whole event, as listed on Data collected.
-- Prefix and slash runs per command this week (MySQL; PostgreSQL uses data->>'command')
SELECT type, JSON_UNQUOTE(JSON_EXTRACT(data, '$.command')) AS command, count(*) AS runs
FROM shardlens_events
WHERE type IN ('interaction', 'message_command') AND ts > NOW() - INTERVAL 7 DAY
GROUP BY 1, 2 ORDER BY runs DESC;
// The same in MongoDB
db.shardlens_events.aggregate([
{ $match: { type: { $in: ['interaction', 'message_command'] }, ts: { $gt: new Date(Date.now() - 7 * 864e5) } } },
{ $group: { _id: { type: '$type', command: '$command' }, runs: { $sum: 1 } } },
{ $sort: { runs: -1 } },
]);

The copy follows the same rules as the collector:

  • Users are the same salted hashes, never user IDs. Nothing the data register marks as identifying is copied: in audit mode, who ran a command stays in the collector’s encrypted store, and your app team’s IDs (used for dashboard sign-in) stay in the collector.
  • Erasure reaches the copy: forgetUser(), /privacy erase and the dashboard’s erase delete from your database the same rows they delete in the collector (in privacy mode, a person’s rows under the salts the bot still holds: the current and previous period), and the request fails (so it is retried) if your database can’t be reached or hasn’t answered within 20 seconds.
  • Copies are deleted after SHARDLENS_SINK_RETENTION_DAYS. If you keep them longer than about a month, their hashes outlive the salt that made them: they can no longer be tied to anyone, or erased by user.
  • Message content is never collected, so it is never copied.

Your privacy policy should name the database you copy to. npx shardlens privacy prints a paragraph to start from.

  • Only events accepted from the moment the sink is turned on are copied. Nothing is backfilled.
  • Copies wait in memory (up to 20,000 events) while your database is down, and are dropped and counted after that. The collector’s own store is unaffected.
  • If your database keeps refusing a batch while it is reachable (a column too narrow, a strict-mode error), the rows are tried one by one and only those it refuses are dropped and logged, so one bad row never holds up the rest. A sink you write yourself without init counts as unreachable while it refuses every row, so nothing is dropped during an outage.
  • A query to MySQL, MariaDB or PostgreSQL that gets no answer for 30 seconds (a network partition, say) fails and is retried, so a database that stops answering can’t hold the copies up for ever. MongoDB’s driver notices an unreachable server by itself.
  • Webhook events from Discord (installs and removals sent to the collector directly) are not copied.