Troubleshooting
Start with doctor from your bot’s folder. It checks the things below in order:
npx shardlens doctor --url https://stats.example.com --key slk_ingest_…No data on the dashboard
Section titled “No data on the dashboard”- Is the SDK attached before login? Call
createShardlens(client, …)beforeclient.login(). - Is it sending? Read the bot’s own log first: the SDK prints one line starting with
Shardlens:per problem (it can’t reach the collector, the key was rejected, the URL isn’t HTTPS). Then checklens.stats():readyshould betrue. If it staysfalse, the SDK never got its settings from the collector, which means a wrong URL or a wrong key;failedFlushesdoes not move in that case, because nothing was ever sent. Oncereadyistrue,failedFlushesshould not keep growing andqueueDepthshould stay low. - Is it a dry run?
dryRun: trueprints events instead of sending them. - Has anything happened yet? The Overview shows interactions. Run a command in a server with the bot.
- Is the time range right? Charts use the time zone in Settings → General.
401 from the collector
Section titled “401 from the collector”The key was not accepted.
- Use the ingest key (
slk_ingest_…) in your bot. Read keys can’t send events. - Check for a revoked key under Settings → API keys.
- Check the bot’s environment has the full key, with no quotes or whitespace around it.
403 from the collector
Section titled “403 from the collector”The batch is for a different application. SHARDLENS_APP_ID on the collector has to match the bot’s application ID.
From your own dashboard, 403 with activity_key_required means a read key asked for who ran what: that takes an
activity key. 404 with audit_mode_off means the key is
right and the collector isn’t in audit mode.
The collector’s address shows "dashboard":"off"
Section titled “The collector’s address shows "dashboard":"off"”The collector was set up without the Shardlens dashboard (SHARDLENS_DASHBOARD=off, from shardlens init --dashboard own or none), so it answers as an API only. Set SHARDLENS_DASHBOARD=on in its .env and restart it to get the
dashboard back: Choose your setup.
“Salt missing” or events waiting
Section titled ““Salt missing” or events waiting”The SDK hashes user IDs with a salt from the collector’s /v1/config. Until it has a salt, it holds events for up to 60
seconds (1,000 events at most) and then drops them, counting the drops. If this keeps happening:
- Check the bot can reach the collector (
doctorreports Salt available). - Check the collector’s logs for database errors: salts are created by the collector.
413 or 503 from the collector
Section titled “413 or 503 from the collector”413: a batch was too large. The SDK splits batches automatically; if you see this, lowermaxBatchSize.503: the collector is busy. The SDK waits forRetry-Afterand retries, so no events are lost while the queue has room.
Alerts or reports don’t arrive
Section titled “Alerts or reports don’t arrive”The alert history (Alerts) and the Last sent column of Reports say why:
discord returned 404: the webhook was deleted. Create a new one and save the alert or report again.could not reach Discord (…): the collector can’t connect todiscord.com. The code says why, for exampleENOTFOUND(no DNS) orECONNREFUSED; a firewall that only allows inbound traffic is the usual cause.The saved webhook could not be decrypted:SHARDLENS_SETTINGS_KEYchanged. Save the webhook again.
Webhook events stopped
Section titled “Webhook events stopped”Discord turns webhook events off when the endpoint keeps failing.
- Check
DISCORD_PUBLIC_KEYmatches General Information → Public Key. - Check
https://stats.example.com/v1/discord/eventsis reachable from the internet. - In the Developer Portal, save the endpoint URL again and turn events back on.
Can’t sign in with Discord
Section titled “Can’t sign in with Discord”- The redirect in the Developer Portal must be exactly
https://stats.example.com/auth/discord/callback, matchingSHARDLENS_PUBLIC_URL. - Only the application owner, team members and IDs in
SHARDLENS_EXTRA_ADMINSget in. Others can use the read key.
Collector won’t start
Section titled “Collector won’t start”The log names each invalid variable. The usual causes are a short SHARDLENS_SESSION_SECRET (32 characters minimum),
a SHARDLENS_SETTINGS_KEY that isn’t 32 base64 bytes, or only one of DISCORD_CLIENT_ID and DISCORD_CLIENT_SECRET.
Reading the collector’s logs
Section titled “Reading the collector’s logs”The collector writes one JSON object per line (docker compose logs collector). Every line has a level (30 info,
40 warning, 50 error), a time and a msg. For a readable view, pipe them through
pino-pretty: docker compose logs -f --no-log-prefix collector | npx pino-pretty.
| Message | Level | What it means |
|---|---|---|
Shardlens collector started |
info | Startup finished. Lists the database migrations this start applied (migrationsApplied, empty when the database was up to date), which optional features are switched on (Discord sign-in, webhook events, audit key, ClickHouse, the sink to your own database), whether the dashboard is served, and the mode users are recorded in (mode, and modeFrom: environment when SHARDLENS_MODE set it). |
Server listening at … |
info | The address the collector accepts connections on. |
shutting down |
info | The collector received SIGTERM or SIGINT and is finishing what it was doing. |
Demo mode: … |
warning | SHARDLENS_DEMO is on: the dashboard is open to anyone. For demo data only. |
Live tiles are not shared … |
warning | PostgreSQL refused LISTEN at startup, so with several collectors each one’s live tiles show only its own traffic until a retry in the background succeeds. |
Live tiles lost their LISTEN connection … |
warning | The connection that shares live tiles broke (a database restart, say). The collector subscribes again, waiting up to 30 seconds between tries. |
this collector is now the job leader |
info | This instance runs the background jobs (rollups, alerts, retention). Exactly one instance does. |
events rejected |
warning | A batch was accepted but some events were dropped. reasons counts them: too_old (older than your raw-data retention: 35 days unless Settings says otherwise), in_future (more than a day ahead), unknown_salt_period (hashed with a salt that was destroyed), invalid_event, refused_by_database (a value PostgreSQL would not store). |
input refused by the database |
warning | A request carried a value PostgreSQL would not store. It was answered with 400; nothing was saved from it. |
clock skew corrected for batch |
warning | The bot’s clock is more than 5 minutes off. Timestamps were corrected; fix the clock (NTP). |
Discord sign-in failed |
warning | The OAuth exchange failed. reason names the error, usually a wrong client secret or redirect URL. |
Discord sign-in refused: … |
warning | Someone who is not the application owner, a team member or an extra admin tried to sign in. If it says the bot has not reported yet, start the bot with Shardlens once, or add your user ID to SHARDLENS_EXTRA_ADMINS. |
No ingest key: … |
warning | No ingest key exists, in the environment or created in the dashboard. Set SHARDLENS_INGEST_KEY. |
No way to open the dashboard: … |
warning | Neither a read key nor Discord sign-in is set up. Set SHARDLENS_READ_KEY. |
No read key, and the dashboard is off: … |
warning | SHARDLENS_DASHBOARD=off and no read or activity key exists, so neither your own dashboard nor a script can read the numbers. Set SHARDLENS_READ_KEY. |
Discord sign-in works once … |
warning | Only Discord sign-in is set up and the bot hasn’t reported yet. Set a read key or SHARDLENS_EXTRA_ADMINS meanwhile. |
live update failed |
warning | The live tiles could not read the database. Logged once per open dashboard. |
slow job |
info | A background job took more than 10 seconds. Frequent on a healthy database means it needs more resources. |
request failed |
error | A request hit an unexpected error. reqId matches the requestId in the 500 response the caller received. |
job failed |
error | A background job threw. job names it; it runs again on its next schedule. partitions in use means a long query kept a table locked for 2 seconds while retention dropped old data: it tries again a minute later instead of holding ingest up. |
rows the database could not aggregate were skipped |
error | A rollup statement refused its data (a sum out of range). skipped names the statement and window; every other metric was still rolled up. |
webhook event processing failed |
error | A Discord webhook event was acknowledged but could not be stored. |
<database> sink … failed: … |
warning | The copy of events in your own database could not be set up or written (component: sink). Logged at most once a minute, with any password removed; copies wait in memory and are retried. |
<database> sink is behind: … |
warning | Your database was down or slow for long enough that 20,000 copies were waiting, so newer ones were dropped. The count is in the message. |
ClickHouse insert failed … |
warning | The ClickHouse copy could not be written (component: clickhouse). Rows wait in memory and are retried; the count of dropped rows is in the message. |
leader election failed |
error | This instance could not check the job lock. It retries every 30 seconds. |
Logs never contain keys, tokens, cookies, request bodies or user IDs. Per-request lines (“incoming request”) contain
URLs, so they are only written when LOG_LEVEL is debug or trace. That includes requests the collector turned
away: a 401 for a wrong key, a 403, a 429 for a caller over its limit. At the default level those are not logged
at all, so a bot with a wrong key shows up in the bot’s log (Shardlens: the collector rejected the API key), not in
the collector’s.
In your bot, the SDK logs through console (or the logger you pass) with a Shardlens: prefix, once per problem
rather than once per event, and lens.on('error', …) and lens.stats() expose everything it counted.