Skip to content

LGTM stack

A separate Compose stack at observability/docker-compose.lgtm.yml provides metrics, logs, and traces. It is independent of the trading stack and can run on the same host or be pointed at a remote one.

| Container | Image | Port (dev container) | Role | | --- | --- | --- | --- | | lgtm-grafana | grafana/grafana:11.4.0 | 3000 | Dashboards, alerting, datasource UI | | lgtm-prometheus | prom/prometheus:v3.1.0 | 9090 | Time-series metrics; remote-write enabled | | lgtm-loki | grafana/loki:3.3.2 | 3100 | Log aggregation | | lgtm-tempo | grafana/tempo:2.7.1 | 3200 / 4319 | Distributed traces | | lgtm-alloy | grafana/alloy:v1.7.0 | 12345 | Log + metric collector; tails service stdout |

Grafana is provisioned with the three datasources and a Trading folder containing both the project-specific dashboards and the official k6 Prometheus dashboard.

Terminal window
docker compose -f observability/docker-compose.lgtm.yml up -d

In the dev container, open http://localhost:3000 once the stack is up. In production, Grafana is reachable at https://veta.mnetcs.com/grafana/. Anonymous read access is enabled; admin and edit access remain auth-gated.

Grafana is auto-provisioned with the following dashboards under the Trading folder. They live in observability/grafana/provisioning/dashboards/ and update on disk-change without a Grafana restart.

| Dashboard | Source | What it shows | | --- | --- | --- | | Trading Overview | Loki via kafka-relay | Order pipeline counters, fill rate, algo heartbeat, service versions, live event feeds. Works without OTel. | | Service Performance (OTel) | Tempo span-metrics + Loki | Request/error rate and p50/p95/p99 latency per service, Top-N slowest/busiest spans, scoped log panel. | | Order Pipeline (Traces) | Tempo + Loki | Order lifecycle trace view: per-stage p95 latency, live service-graph nodeGraph, recent error/slow-trace tables. | | k6 Prometheus | Prometheus | Bundled k6 dashboard for load-test runs. |

The two OTel dashboards depend on the compose.observability.yml overlay being active. Without it there are no spans to render.

The screenshots below are captured directly from the production Grafana via its image-renderer by docs/site/scripts/capture-grafana-screenshots.mjs. To refresh, run npm run capture:grafana from docs/site/ against a host with the LGTM stack up. Click any panel for the full-resolution version.

Captured 75 days ago from the homelab LGTM stack — .

Order Pipeline (Traces)

Trace-driven view of the order lifecycle. The platform's HTTP and Kafka hops both propagate trace context, so a single submitted order produces one connected trace from Gateway through OMS, Risk, Algo, EMS, and Journal. This dashboard surfaces the slowest, the failed, and the most active.

Order submission latency (gateway span p50/p95/p99)
Order submission latency (gateway span p50/p95/p99)Latency of the gateway's order-submission span — what the user actually waited for.
Per-stage latency (p95)
Per-stage latency (p95)p95 latency for each pipeline service. Stages with rising lines are bottlenecks.
Order events from bus
Order events from busOrder lifecycle events as observed by kafka-relay → Loki. Use this when traces are too detailed.

Service Performance (OTel)

Per-service request rate, latency, errors, and resource usage — sourced from Deno's built-in OpenTelemetry instrumentation. All panels respect the service variable, so pick a service from the top to scope the view.

Error rate by service
Error rate by serviceErrors per second per service.
HTTP request body size by service
HTTP request body size by serviceBytes received per second on inbound HTTP, broken out by service. Useful for spotting unusually chatty clients.
Inter-service request rate
Inter-service request rateEdges of the service graph — who calls whom, how often. Sourced from Tempo's service-graph processor.
Inter-service p95 latency
Inter-service p95 latencyp95 latency for each call-pair in the service graph — useful for tracking which downstream calls dominate end-to-end latency.

Trading Overview

Service Health
Service Health
Order Pipeline — Submitted / Filled / Rejected / Expired
Order Pipeline — Submitted / Filled / Rejected / Expired
HTTP Request Rate (req/s per service)
HTTP Request Rate (req/s per service)
HTTP Latency p99 (all services)
HTTP Latency p99 (all services)
Algo Heartbeat Rate (beats/min per strategy)
Algo Heartbeat Rate (beats/min per strategy)
Order Funnel (selected window)
Order Funnel (selected window)Counts of orders at each lifecycle stage over the selected time window
Fill Rate % (last 5 min)
Fill Rate % (last 5 min)
Child Order & Fill Throughput (per min)
Child Order & Fill Throughput (per min)
Process Resident Memory (MB) per service
Process Resident Memory (MB) per service