Mission Control

When something is wrong, three pieces of information are needed: what is broken, where to look, and what to do about it. This page is the index. Everything below is a hop, not a hub.

Note

These diagnostic tools are operator-only and live on a private network. URLs and access instructions are kept in the team password manager rather than in public docs.

Live diagnostics (operator access)

The platform uses the standard Grafana LGTM stack for live observability (Grafana for dashboards, Tempo for traces, Loki for logs), plus Traefik's built-in dashboard for routing state, Redpanda Console for Kafka topic inspection, and Uptime Kuma for external health checks.

These run inside the deployment's private network and aren't exposed publicly. Operator-only credentials and URLs are in the team password manager.

In-app tools

Logs drawer (in trading UI) Header to Logs button. Loki when it's reachable, gateway in-process ring buffer when it isn't.

Service Health panel Live status of every backend service. Add via + Add Panel, then Service Health.

Decision Log panel Algo decisions in real time. Useful for understanding why a child order was or wasn't created.

Estate Overview Single-pane view of services, throughput and active alerts. Good for screen-sharing during incidents.

Common runbooks

A service is restart-looping Recent crash-on-boot pattern: missing DATABASE_URL, port collision, dependency not yet healthy.

A new service didn't deploy The systemd auto-pull only re-runs deploy.sh when the main branch SHA changes. A force-deploy is `sudo systemctl start veta-auto-pull.service`.

Login is broken user-service is the failure surface. Check Postgres, validate the cookie, look at gateway to user-service traces.

Market ladder shows no prices Market-sim publishing? Gateway WebSocket connected? Frontend got 'authIdentity'? Walk the chain in this order.

Algo regression Save a baseline scenario, run it three times. Identical fills = good. Drift = bug.

Service restart loop

Read the failing service's logs first via Grafana, Loki, {service_name="<svc>"}. Common causes ordered by frequency:

1. Missing env var. Most often a database URL or service host. Compare the service's compose.yml block to its consumer module.
2. Schema migration not yet applied. The service depends on db-migrate completing. Check docker logs veta-db-migrate-1: exit 0 means done; anything else means a migration failed.
3. Dependency not healthy. A depends_on: condition: service_healthy will hold the dependent service back. The unhealthy upstream is where the real problem is.
4. Port collision. Rare in compose. Check docker ps for duplicate names.

Note: if the service has been deleted manually, the next auto-pull tick (every 5 min) will run deploy.sh, which brings the full compose stack back up. A restart loop cannot be papered over by stopping the container; fix the underlying issue.

Deploy and CI

GitHub Actions, main branch CI run for the latest commit. Should be green within ~12 minutes of a push.

Container registry Every service has its own GHCR image. The server auto-pull picks up :latest within 5 minutes of CI publishing it.

Release Please PRs Auto-generated version bumps. Merging promotes whatever's on main to a tagged release.

Reference

Glossary Searchable index of every term, acronym and service name. Dangling references fail the docs build.

Service map Every service, its port, its purpose, its repo path. Auto-generated from the codebase.

Architecture overview Order-flow diagram, bus topics, end-to-end picture of the platform.

API gateway endpoints Every public route the gateway serves, who's allowed to call it, and what shape the response takes.