NVIDIA Spark (LLM host)

The production environment has a dedicated NVIDIA Spark workstation that hosts a local Ollama instance. Inference workloads that previously required external API calls (OpenAI, Anthropic, Groq) now stay on the LAN.

Network position

Host: <gpu-host> (separate machine from the server proper at <server-host>)
Ollama API: http://<gpu-host>:11434 (raw API, no auth on the LAN)
Open WebUI: http://<gpu-host>:3000 (interactive UI, auth required)

The Spark is on the same LAN as the server so latency is single-digit milliseconds. Traffic does not traverse the edge server.

Models loaded

| Model | Quant | Size | Best for | | --- | --- | --- | --- | | qwen3-coder:30b | Q4_K_M | 19 GB | Code edits, agent loops, structured outputs | | deepseek-r1:70b | Q4_K_M | 43 GB | Chain-of-thought planning, harder reasoning | | gpt-oss:120b | MXFP4 | 65 GB | Heavy reasoning, longest-context jobs | | llama2-uncensored:70b | Q4_0 | 39 GB | Legacy; rarely the right tool |

qwen3-coder:30b is the default recommendation for VETA's LLM-advisory service because it balances quality and VRAM footprint, and it is tuned for code-context generation.

Pointing LLM-advisory at the Spark

The advisory service reads its base URL and model from environment variables, both of which fall through to compose defaults. Override either on the server.

In /opt/stacks/veta/.env:

LLM_OLLAMA_BASE_URL=http://<gpu-host>:11434
LLM_MODEL_ID=qwen3-coder:30b

Then restart the consumers so they pick up the new config:

cd /opt/stacks/veta
docker compose up -d llm-advisory llm-advisory-worker

The dev-container compose file keeps http://ollama:11434 as the default so a developer running the local stack still gets the small qwen2.5:3b model and does not depend on the Spark being reachable.

Smoke-testing

Direct probe of the Ollama API:

curl -sS http://<gpu-host>:11434/api/tags | jq '.models[].name'

End-to-end check of LLM-advisory:

curl -sS -X POST http://veta.home/api/llm-advisory/advisories \
  -H 'Content-Type: application/json' \
  --cookie "$YOUR_SESSION_COOKIE" \
  -d '{"symbol":"AAPL"}'

When not to use it

Anything that needs sub-second latency at scale (the 70-120B models can take 2-5s per response under load)
Workloads that run inside compose.yml's trading-net Docker network but cannot reach the server LAN (none currently)
Tasks where the response should be reproducible across deployments — the Spark model set is environment-specific

Future uses

Parallel-agent work: a local agent pulling tickets from a queue and running qwen3-coder:30b against a worktree of this repo to draft patches on a side branch
Monte Carlo and vol-surface fitting offload from the CPU-bound analytics service
Heavy backtests