Alert routing

VETA routes two classes of alerts to the same Discord channel:

• Platform alerts from Grafana, driven by Prometheus rules. Memory pressure, breaker activity, disk-fill warnings, service-down events.
• User-level alerts from the live trading UI, driven by alertsMiddleware in the frontend. Kill-switch activations, order rejections by the risk engine, workspace save failures.

Both classes hit the same webhook URL so a single channel is the on-call notification surface.

Flow

flowchart LR
  subgraph Frontend
    A1[alertAdded action] --> A2[alertsMiddleware]
    A2 --> A3[POST /api/gateway/alerts]
  end

  subgraph Gateway
    A3 --> G1[handleAlertsRoute]
    G1 --> G2[notifyDiscord]
    G1 --> G3[forward to user-service]
  end

  subgraph Observability
    P[Prometheus rules] --> GR[Grafana AlertManager]
    GR --> CP[contact-point: veta-alerts]
  end

  G2 --> DC[Discord webhook]
  CP --> DC
  DC --> CH[(#veta-alerts channel)]

Severity filtering

The gateway only fans CRITICAL and WARNING out to Discord. INFO alerts (workspace-save failures, routine notifications) stay inside the in-app alert centre and are not surfaced to ops.

Grafana applies its own routing by category label: anything matched by category=security goes to the legacy security-webhook contact point so security alerts can be split out later if needed.

Ticketing for CRITICAL alerts

CRITICAL alerts also auto-create a GitHub issue. The gateway's alerts route runs notifyDiscord and createTicketForAlert side-by-side via Promise.allSettled, so a Discord-delivery failure can't block ticket creation and vice versa.

The issue body captures severity, source, message, detail, timestamp, and the user who triggered it. Labels: prod-issue, auto-created, severity:critical, plus source:<name> when the alert names one.

Duplicate suppression: before creating an issue, the gateway searches for an open issue with the same title created within the last hour. If one exists, it adds a comment to that issue instead of opening a duplicate. This keeps recurring alerts (e.g. a flapping service) on one ticket.

See the ticketing runbook for env wiring, token scopes, and end-to-end verification.

Configuration

The webhook URL is injected at runtime via the DISCORD_WEBHOOK_URL environment variable. It is never committed to the repository.

| Service | Env var | Compose default | | --- | --- | --- | | gateway | DISCORD_WEBHOOK_URL | empty string (notifier no-ops) | | grafana | DISCORD_WEBHOOK_URL | https://discord.com/api/webhooks/0/REPLACE_ME (rejected by notifier) |

On the server, set the real URL in /opt/stacks/veta/.env:

DISCORD_WEBHOOK_URL=https://discord.com/api/webhooks/<id>/<token>

Then restart the consuming containers so they pick up the new value:

cd /opt/stacks/veta
docker compose up -d gateway
docker compose -f observability/docker-compose.lgtm.yml up -d grafana

Verifying delivery

Smoke-test the webhook directly:

curl -sS -i -X POST \
  -H "Content-Type: application/json" \
  -d '{"username":"VETA Alerts","content":"smoke test"}' \
  "$DISCORD_WEBHOOK_URL"

A working webhook returns HTTP/2 204 with empty body. A revoked or malformed URL returns HTTP/2 401 or 404.

End-to-end test through the platform:

1. As an admin, trigger the kill switch from the Mission Control panel
2. Watch the alert centre populate within ~200ms
3. Watch Discord receive the formatted message within ~1s

Rotating the webhook

Discord webhook URLs are bearer credentials. Rotate immediately if the URL appears in chat logs, screenshots, or any context where it might leak.

1. Discord channel settings → Integrations → Webhooks → click the existing webhook
2. Either delete and recreate (creates a fresh URL) or copy the URL and regenerate the token from the same panel
3. SSH to the server and update /opt/stacks/veta/.env
4. docker compose up -d gateway grafana to apply
5. Smoke-test with the curl one-liner above
6. The old URL becomes inert immediately; no further cleanup needed

Webhook tokens are bearer credentials

Anyone with the webhook URL can post arbitrary messages to the channel as VETA Alerts. Treat the URL like an API key. Never commit it, never paste it in PR descriptions, and rotate after any exposure.

Daily summary

Once a day at 09:00 UTC, the gateway posts a 24h rollup of platform health to the same Discord webhook. Override the hour via DISCORD_DAILY_SUMMARY_HOUR_UTC (valid range 0–23; invalid values fall back to 9).

The message shape:

✅ **VETA daily summary** · `a1b2c3d` (prod) · gateway uptime 2d 14h 3m

**Services (last 24h):** mean 99.8% up · worst window 96.9%
Now: 32/32 up · ✅

**Alerts (last 24h):** 7 total · critical: 1, warning: 6
Last critical: 47m ago — `kill-switch` admin halt for AAPL

**Bug reports (last 24h):** 2 from 2 users

**Deployed SHA:** `a1b2c3d`

Header emoji follows the worst service-availability window seen in the day:

| Emoji | Worst-window service ratio | | --- | --- | | ✅ | 100% (no service ever dropped) | | ⚠️ | 95% ≤ ratio < 100% | | 🚨 | below 95% |

So ✅ means the platform was fully healthy every 5-second sample during the day. Drop below 100% even briefly and the rollup downgrades.

Implementation in backend/src/gateway/daily-summary.ts, backed by the in-memory ring buffer in platform-stats.ts. Events older than 24h are pruned on every snapshot read. No background polling — service availability is sampled from the same refreshHealth loop the gateway already runs.

Replaces the per-minute heartbeat that landed in #319 — 1440 messages/day buried real alerts.

CI/CD notifications

Three GitHub workflow events fire into the same channel:

| Event | Trigger | Message format | | --- | --- | --- | | CI failure on main | workflow_run.conclusion == 'failure' | 🚨 <workflow> failed on main · <sha> by <actor> | | Docs deploy success | Deploy GitHub Pages completed on main | 📘 Docs deployed · <sha> | | PR merged to main | pull_request_target.merged == true | ✅ Merged #N — <title> (by <author>) |

Bot-authored merges (release-please, dependabot, github-actions[bot]) are filtered out so the channel doesn't fill with screenshot/badge auto-commits.

Wired in .github/workflows/notify-discord.yml. The workflow reads secrets.DISCORD_WEBHOOK_URL — set the same webhook URL as a repo secret. If the secret is unset, every job logs "skipping" and exits 0, so the workflow is safe to land before the secret is wired up.

Frontend memory metric

The gateway emits frontend_memory_heap_used_max_bytes as an OTEL gauge — the max jsHeapSizeUsed across all live browser sessions reporting via the useFrontendMemoryTelemetry hook. The metric flows via the existing OTLP plumbing → Alloy → Prometheus.

Two alert rules consume it:

| Rule | Threshold | Duration | Severity | | --- | --- | --- | --- | | FrontendHeapHigh | > 2 GiB | 10m | warning | | FrontendHeapCritical | > 4 GiB | 2m | critical |

Hitting Critical typically precedes a frozen tab (Firefox starts swapping at 4 GiB). Get the affected user to hard-refresh and grab a heap snapshot from about:memory if you want to investigate further.

The metric is per-gateway (one value for "max across all users") rather than per-user. Cardinality stays bounded regardless of user count, but specific-user tracing requires looking at the raw POSTs to /telemetry/frontend.

Adding new alert rules

Backend Prometheus rules live in observability/prometheus-rules.yml. They auto-load when Grafana boots; no migration step.

Every rule has a promtool unit test in observability/prometheus-rules.test.yml. Tests cover the positive case (rule fires when threshold is breached for the configured for: window) and the locked-in negatives (no false positives below the threshold, market.* topics excluded from the topic-rate rules, per-group fan-out for consumer lag). Run them locally:

promtool test rules observability/prometheus-rules.test.yml

Or via docker if promtool is not installed:

docker run --rm \
  -v "$PWD/observability/prometheus-rules.yml":/prometheus-rules.yml \
  -v "$PWD/observability/prometheus-rules.test.yml":/test.yml \
  --entrypoint promtool prom/prometheus:v3.1.0 test rules /test.yml

Pre-commit runs these automatically when either file is staged. CI runs them on every PR that touches either file.

User-level alerts are dispatched by alertsMiddleware reacting to Redux actions. Add a new branch to the middleware to emit on additional event types. Severity ≥ WARNING reaches Discord automatically; CRITICAL also opens a GitHub ticket.

Troubleshooting

• No messages in Discord, but the alert appears in-app: check the gateway has DISCORD_WEBHOOK_URL set. docker compose exec gateway env | grep DISCORD.
• Grafana alerts not arriving but Prometheus rules are firing: check Grafana → Alerting → Contact points → veta-alerts and click "Test". If that fails, the webhook URL is wrong or revoked.
• INFO alerts not appearing: by design. Only CRITICAL/WARNING fan out. Change notifyDiscord severity filter if you want broader coverage.
• Channel is silent during a known incident: confirm the rule's for: duration. ServiceMemoryGrowthSustained requires 15 minutes of sustained growth before firing.