Skip to content

Troubleshooting

Start with doctor from your bot’s folder. It checks the things below in order:

Terminal window
npx shardlens doctor --url https://stats.example.com --key slk_ingest_…
  1. Is the SDK attached before login? Call createShardlens(client, …) before client.login().
  2. Is it sending? Read the bot’s own log first: the SDK prints one line starting with Shardlens: per problem (it can’t reach the collector, the key was rejected, the URL isn’t HTTPS). Then check lens.stats(): ready should be true. If it stays false, the SDK never got its settings from the collector, which means a wrong URL or a wrong key; failedFlushes does not move in that case, because nothing was ever sent. Once ready is true, failedFlushes should not keep growing and queueDepth should stay low.
  3. Is it a dry run? dryRun: true prints events instead of sending them.
  4. Has anything happened yet? The Overview shows interactions. Run a command in a server with the bot.
  5. Is the time range right? Charts use the time zone in Settings → General.

The key was not accepted.

  • Use the ingest key (slk_ingest_…) in your bot. Read keys can’t send events.
  • Check for a revoked key under Settings → API keys.
  • Check the bot’s environment has the full key, with no quotes or whitespace around it.

The batch is for a different application. SHARDLENS_APP_ID on the collector has to match the bot’s application ID.

From your own dashboard, 403 with activity_key_required means a read key asked for who ran what: that takes an activity key. 404 with audit_mode_off means the key is right and the collector isn’t in audit mode.

The collector’s address shows "dashboard":"off"

Section titled “The collector’s address shows "dashboard":"off"”

The collector was set up without the Shardlens dashboard (SHARDLENS_DASHBOARD=off, from shardlens init --dashboard own or none), so it answers as an API only. Set SHARDLENS_DASHBOARD=on in its .env and restart it to get the dashboard back: Choose your setup.

The SDK hashes user IDs with a salt from the collector’s /v1/config. Until it has a salt, it holds events for up to 60 seconds (1,000 events at most) and then drops them, counting the drops. If this keeps happening:

  • Check the bot can reach the collector (doctor reports Salt available).
  • Check the collector’s logs for database errors: salts are created by the collector.
  • 413: a batch was too large. The SDK splits batches automatically; if you see this, lower maxBatchSize.
  • 503: the collector is busy. The SDK waits for Retry-After and retries, so no events are lost while the queue has room.

The alert history (Alerts) and the Last sent column of Reports say why:

  • discord returned 404: the webhook was deleted. Create a new one and save the alert or report again.
  • could not reach Discord (…): the collector can’t connect to discord.com. The code says why, for example ENOTFOUND (no DNS) or ECONNREFUSED; a firewall that only allows inbound traffic is the usual cause.
  • The saved webhook could not be decrypted: SHARDLENS_SETTINGS_KEY changed. Save the webhook again.

Discord turns webhook events off when the endpoint keeps failing.

  1. Check DISCORD_PUBLIC_KEY matches General Information → Public Key.
  2. Check https://stats.example.com/v1/discord/events is reachable from the internet.
  3. In the Developer Portal, save the endpoint URL again and turn events back on.
  • The redirect in the Developer Portal must be exactly https://stats.example.com/auth/discord/callback, matching SHARDLENS_PUBLIC_URL.
  • Only the application owner, team members and IDs in SHARDLENS_EXTRA_ADMINS get in. Others can use the read key.

The log names each invalid variable. The usual causes are a short SHARDLENS_SESSION_SECRET (32 characters minimum), a SHARDLENS_SETTINGS_KEY that isn’t 32 base64 bytes, or only one of DISCORD_CLIENT_ID and DISCORD_CLIENT_SECRET.

The collector writes one JSON object per line (docker compose logs collector). Every line has a level (30 info, 40 warning, 50 error), a time and a msg. For a readable view, pipe them through pino-pretty: docker compose logs -f --no-log-prefix collector | npx pino-pretty.

Message Level What it means
Shardlens collector started info Startup finished. Lists the database migrations this start applied (migrationsApplied, empty when the database was up to date), which optional features are switched on (Discord sign-in, webhook events, audit key, ClickHouse, the sink to your own database), whether the dashboard is served, and the mode users are recorded in (mode, and modeFrom: environment when SHARDLENS_MODE set it).
Server listening at … info The address the collector accepts connections on.
shutting down info The collector received SIGTERM or SIGINT and is finishing what it was doing.
Demo mode: … warning SHARDLENS_DEMO is on: the dashboard is open to anyone. For demo data only.
Live tiles are not shared … warning PostgreSQL refused LISTEN at startup, so with several collectors each one’s live tiles show only its own traffic until a retry in the background succeeds.
Live tiles lost their LISTEN connection … warning The connection that shares live tiles broke (a database restart, say). The collector subscribes again, waiting up to 30 seconds between tries.
this collector is now the job leader info This instance runs the background jobs (rollups, alerts, retention). Exactly one instance does.
events rejected warning A batch was accepted but some events were dropped. reasons counts them: too_old (older than your raw-data retention: 35 days unless Settings says otherwise), in_future (more than a day ahead), unknown_salt_period (hashed with a salt that was destroyed), invalid_event, refused_by_database (a value PostgreSQL would not store).
input refused by the database warning A request carried a value PostgreSQL would not store. It was answered with 400; nothing was saved from it.
clock skew corrected for batch warning The bot’s clock is more than 5 minutes off. Timestamps were corrected; fix the clock (NTP).
Discord sign-in failed warning The OAuth exchange failed. reason names the error, usually a wrong client secret or redirect URL.
Discord sign-in refused: … warning Someone who is not the application owner, a team member or an extra admin tried to sign in. If it says the bot has not reported yet, start the bot with Shardlens once, or add your user ID to SHARDLENS_EXTRA_ADMINS.
No ingest key: … warning No ingest key exists, in the environment or created in the dashboard. Set SHARDLENS_INGEST_KEY.
No way to open the dashboard: … warning Neither a read key nor Discord sign-in is set up. Set SHARDLENS_READ_KEY.
No read key, and the dashboard is off: … warning SHARDLENS_DASHBOARD=off and no read or activity key exists, so neither your own dashboard nor a script can read the numbers. Set SHARDLENS_READ_KEY.
Discord sign-in works once … warning Only Discord sign-in is set up and the bot hasn’t reported yet. Set a read key or SHARDLENS_EXTRA_ADMINS meanwhile.
live update failed warning The live tiles could not read the database. Logged once per open dashboard.
slow job info A background job took more than 10 seconds. Frequent on a healthy database means it needs more resources.
request failed error A request hit an unexpected error. reqId matches the requestId in the 500 response the caller received.
job failed error A background job threw. job names it; it runs again on its next schedule. partitions in use means a long query kept a table locked for 2 seconds while retention dropped old data: it tries again a minute later instead of holding ingest up.
rows the database could not aggregate were skipped error A rollup statement refused its data (a sum out of range). skipped names the statement and window; every other metric was still rolled up.
webhook event processing failed error A Discord webhook event was acknowledged but could not be stored.
<database> sink … failed: … warning The copy of events in your own database could not be set up or written (component: sink). Logged at most once a minute, with any password removed; copies wait in memory and are retried.
<database> sink is behind: … warning Your database was down or slow for long enough that 20,000 copies were waiting, so newer ones were dropped. The count is in the message.
ClickHouse insert failed … warning The ClickHouse copy could not be written (component: clickhouse). Rows wait in memory and are retried; the count of dropped rows is in the message.
leader election failed error This instance could not check the job lock. It retries every 30 seconds.

Logs never contain keys, tokens, cookies, request bodies or user IDs. Per-request lines (“incoming request”) contain URLs, so they are only written when LOG_LEVEL is debug or trace. That includes requests the collector turned away: a 401 for a wrong key, a 403, a 429 for a caller over its limit. At the default level those are not logged at all, so a bot with a wrong key shows up in the bot’s log (Shardlens: the collector rejected the API key), not in the collector’s.

In your bot, the SDK logs through console (or the logger you pass) with a Shardlens: prefix, once per problem rather than once per event, and lens.on('error', …) and lens.stats() expose everything it counted.