Observability

VETA emits metrics, logs, and traces to a separate LGTM stack: Loki, Grafana, Tempo, and Mimir/Prometheus, with Grafana Alloy collecting events. The trading stack runs fine without it; bring it up when investigating performance, debugging event flow, or generating data for the Performance page.

Not for a real-bank production

The self-hosted LGTM stack and the OTel auto-instrumentation overlay described in this section are intended for the production demo. A real production deployment should send OTLP to a managed observability vendor (Grafana Cloud, Honeycomb, Datadog, New Relic) rather than running its own collector + storage on the same host as the trading platform. Co-locating the observability backend with the workload it observes creates a noisy-neighbour failure mode where a metric spike crashes both the platform and the dashboard you would have used to diagnose it.

The instrumentation itself (OTEL_DENO=true, the trace-propagating Kafka helpers, the withSpan() calls) works against any OTLP receiver. Only the receiver and storage choice changes.

Sub-pages

LGTM stack Compose layout, provisioned Grafana dashboards, live snapshots from the production.

Log collection and search How events reach Loki via Alloy, the in-app log drawer, and the gateway log-query endpoint.

Distributed tracing OpenTelemetry across HTTP and Kafka hops, the withSpan helper, and pipeline-stage latency.

Alert routing How platform and user-level alerts reach Discord, with the rotation runbook.

Ticketing for prod issues CRITICAL alerts auto-create GitHub issues with dedupe and on-call workflow.

User tickets Users raise bugs, feature requests, and comments; Discord is notified and GitHub Issues tracks tickets when configured.

Discord welcome bot Greets new members on join via the Discord Gateway; bot setup, intents, and rotation.

Session replay rrweb session capture, the player, and how WebM exports are rendered client-side.

Where logs surface

Logs from the platform appear in three places, depending on what's running. Use this decision tree:

| You want to | Use this | What it shows | | --- | --- | --- | | Search recent logs from inside the app, no extra setup | Logs drawer in the app's status bar (Logs button, top-right) | Last ~2,000 lines from the gateway's in-memory ring buffer; falls back to Loki if LOKI_URL is set | | Full historical search across every service | Grafana > Explore > Loki datasource at http://localhost:3000 (dev container; https://veta.mnetcs.com/grafana/ in production) | Everything Alloy has shipped to Loki; backed by the LGTM stack | | Tail a single service's stdout while debugging | supervisorctl tail -f <service> on the host running the platform | Live stdout/stderr from supervisord; same lines that Alloy is collecting | | Track an order through the whole pipeline | Grafana > Tempo > search by trace ID | Distributed trace spanning gateway, OMS, algos, EMS, and journal, with linked log lines | | Per-service container logs (Docker / k8s) | docker logs <container> | Raw stdout from a single service container |

The in-app Logs drawer always works with zero infrastructure, even on a fresh dev container. Each row is filterable by service, log level (error / warn / info / debug), time window (5m to 6h), and free-text search.

When the LGTM stack is up (docker compose -f observability/docker-compose.lgtm.yml up -d) and the gateway has LOKI_URL=http://localhost:3100 set, the same drawer transparently switches to LogQL queries against Loki. Same UI, full history. The drawer's response includes a source: "loki" | "ring-buffer" field so you always know which one answered.

For the in-app drawer's API endpoint, role-gating, and ring-buffer fallback details, see Log collection and search.

Where to run what

| Environment | Trading stack | LGTM stack | k6 | | --- | --- | --- | --- | | local (devcontainer) | yes | optional | optional | | production | yes | yes | manual |

LGTM and k6 are optional locally. They add real overhead and the platform runs fine without them.