# Inference Fleet — System Definition

_This file is the living source of truth for the inference fleet. The interactive atlas is built from the same data — edit this file, rebuild, republish, never hand-edit the outputs._

_Question status: **1 open · 1 resolved**._

## One paragraph

Hermes answers through per-host gateways over a Tailscale tailnet. **Talos primary** is cloud xAI grok-4.6; **straylight primary** is zai glm-5.3-flash. Fallback and delegation go to straylight llama.cpp :11434; small-model aux (approval, titles, web extract) and Honcho compute go to dixie:11434. Six hosts, two local routers, two clouds of compute.

## Decisions locked

| Axis | Decision | ADR |
|---|---|---|
| Primary model | Cloud is the default; local models never carry the main path. Talos primary is xAI grok-4.6. Straylight primary is zai glm-5.3-flash. | — |
| Local router | llama.cpp llama-server on straylight is the main-loop fallback/delegation endpoint. Aux routing (approval/titles/web_extract) is served by dixie:11434. | adr/0001-local-router.md |
| Transport | Mattermost is the gateway transport and hosts the 'claude' overflow user. | — |
| Memory | Honcho runs on rift; its deriver calls dixie:11434 for gen, embed and dialectic. | adr/0002-memory-placement.md |
| GPU ceiling | Straylight (Strix Halo) holds models-max 3 under a ~104 GiB TTM cap. Titan holds models-max 2. Dixie holds models-max 3 on 12 GB. | adr/0003-gpu-ceiling.md |

## Cost model

## Reading order (the atlas chapters)

1. **You and the gateway** — Every turn enters through a gateway, over Mattermost. _(adds gw, mm)_
2. **The primary path** — The default answer comes from the cloud, not from local weights. _(adds xa)_
3. **Local inference** — Fallback, delegation and vision run on the local llama routers. _(adds ll, hl, wp, vl)_
4. **Clients & surfaces** — Every surface — desktop, TUI, browser tools — is a client of the routers. _(adds dk, cp, cd)_
5. **The memory plane** — Memory lives on rift and calls dixie for compute. _(adds hs, oc)_
6. **The fleet** — Four more hosts complete the tailnet — a gateway laptop, a utility NUC, a Metal laptop, a CUDA helper. _(adds tg, or, tn, dx)_
7. **The whole system** — Everything at once — choose a flow and watch a packet walk it.

## Structures

### Cloud APIs

#### Z · zai — api.z.ai

**In one line.** Straylight's primary cloud model.

**What it does.** Nous hosted inference at api.z.ai. Serves glm-5.3-flash, straylight's primary model. Talos does not use this path.

**How it's built.** OpenAI-compatible endpoint. The straylight gateway sends primary turns here.

**Steps in execution.**

1. **Request** — Straylight gateway POSTs the turn.
2. **Model** — glm-5.3-flash generates.
3. **Return** — Streamed back as the primary answer.

#### X · xAI — api.x.ai

**In one line.** Talos's primary cloud model.

**What it does.** xAI hosted inference at api.x.ai. Serves grok-4.6, the model the talos gateway talks to first on every turn.

**How it's built.** OpenAI-compatible endpoint (provider xai-oauth, base_url https://api.x.ai/v1).

**Steps in execution.**

1. **Request** — Talos gateway POSTs the turn.
2. **Model** — grok-4.6 generates.
3. **Return** — Streamed back as the primary answer.

#### C · Anthropic Claude (cloud)

**In one line.** A cloud brain that lives inside Mattermost as the 'claude' user.

**What it does.** Anthropic Claude, reachable only through the Mattermost 'claude' channel. It is the overflow for large-planning tasks that do not fit the local router.

**How it's built.** Not an API the gateway calls directly — it is a Mattermost user. Kyle posts a big planning request to 'claude' in-channel; the reply comes back as a chat message.

**Steps in execution.**

1. **Post** — Send a large-planning request to the claude user.
2. **Reply** — Claude answers in the channel.
3. **Route** — A subagent or Kyle consumes it.

**Questions.**

- **Q-C1** Who reads the claude reply and when does it beat the local router?

#### O · OpenRouter

**In one line.** A multi-model aggregator, currently pointing at a Claude opus mixture.

**What it does.** OpenRouter routes to the moa aggregator claude-opus-4.8. The deepseek reference head is disabled, so it is effectively a single model right now.

**How it's built.** Configured as an alternate cloud path into the gateway (dashed in the diagram). Kept out of the primary flow by policy — billed per-delegation.

**Steps in execution.**

1. **Request** — Optional alternate path.
2. **Aggregate** — moa claude-opus-4.8 (deepseek ref off).
3. **Return** — Answer back to the gateway.

**Questions.**

- ~~**Q-O1** OpenRouter MOA cost ceiling?~~ ✓ Billed per-delegation; kept off the primary path, available on demand (2026-09-21).

### Connection

#### M · Mattermost :443

**In one line.** The transport the gateway rides on, over Tailscale HTTPS.

**What it does.** Mattermost on :443 (Tailscale HTTPS). It carries the gateway transport and hosts the 'claude' user that other agents talk to.

**How it's built.** The Hermes gateway registers as a Mattermost transport; inbound/outbound turns flow through it. Tailscale keeps the port private.

**Steps in execution.**

1. **Inbound** — Turn arrives on the transport.
2. **Forward** — Gateway processes it.
3. **Outbound** — Reply posts back to the channel.

### Gateways

#### G · Hermes gateway (straylight)

**In one line.** Straylight gateway — primary zai, fallback on-box, aux on dixie.

**What it does.** The straylight Hermes gateway. Primary is zai glm-5.3-flash. Fallback and delegation go to on-box llama.cpp :11434. Aux (approval, titles, web extract) goes to dixie:11434.

**How it's built.** Routing lives in the straylight gateway config. Transport is Mattermost; memory calls go to Honcho on rift.

**Steps in execution.**

1. **Dispatch** — Choose primary vs aux by task.
2. **Primary** — Forward to zai glm-5.3-flash.
3. **Fallback/delegation** — Forward to local llama :11434.
4. **Aux** — Small-model aux → dixie :11434.

#### T · Hermes gateway (talos)

**In one line.** Talos gateway plus hermes-serve — no GPU, forwards local work.

**What it does.** On talos (Framework Laptop 12, 13th-gen Intel, no GPU): Hermes gateway and hermes-serve (desktop API/websocket). Primary is xAI grok-4.6. Vision goes to straylight :11434; approval, titles and web extract go to dixie :11434; fallback and delegation go to straylight :11434.

**How it's built.** Talos never runs local gen. custom_providers: llama-router (straylight:11434) and dixie-router (dixie:11434).

**Steps in execution.**

1. **Primary** — xAI grok-4.6.
2. **Vision** — straylight :11434 qwen3-vl-8b.
3. **Aux** — dixie :11434 ornith-1.5-9b.
4. **Fallback/delegation** — straylight :11434 ornith-1.5-35b-a3b.

### straylight — local inference

#### L · llama.cpp router

**In one line.** Main-loop local brain: llama-server for fallback and delegation.

**What it does.** llama.cpp llama-server on straylight :11434, OpenAI-compatible over Vulkan. Main-loop fallback and delegation for the fleet. Small-model aux (approval, titles, web extract) is served by dixie:11434. models-max 3.

**How it's built.** llama-server with Vulkan (gfx1151); TLS via Tailscale Serve :10000. Catalog of about 10 models; up to 3 resident at once under a ~104 GiB TTM cap. ornith-1.5-35b-a3b is the subagent default.

**Steps in execution.**

1. **Catalog** — ~10 models listed; up to 3 resident at a time.
2. **Route** — OpenAI-compatible dispatch by model id.
3. **Serve** — Vulkan inference, models-max 3.

#### H · Honcho CPU lanes

**In one line.** Honcho gen, embed and dialectic — served on dixie.

**What it does.** honcho-gen (ornith-1.5-9b), honcho-embed (Qwen3-Emb-0.6B) and honcho-dialectic (Qwen3-4B-Instruct). These are Honcho's compute; they run as three slots on dixie:11434.

**How it's built.** The Honcho deriver on rift calls dixie:11434 for generation, embedding and dialectic. Not served on straylight.

**Steps in execution.**

1. **gen** — honcho-gen → dixie :11434.
2. **embed** — honcho-embed → dixie :11434.
3. **dialectic** — honcho-dialectic → dixie :11434.

#### W · whisper.cpp

**In one line.** Speech-to-text, CPU-only.

**What it does.** whisper.cpp on :11435 running large-v3-turbo. Local STT for the clients — voice in, text out, no cloud transcription.

**How it's built.** whisper.cpp server, large-v3-turbo, CPU-only. Clients send audio; they get text back before the turn reaches the router.

**Steps in execution.**

1. **Audio** — Client sends audio.
2. **Transcribe** — large-v3-turbo on CPU.
3. **Text** — Return transcript.

#### V · Qwen3-VL-8B (vision)

**In one line.** A dedicated vision model — the router sees images too.

**What it does.** qwen3-vl-8b-abliterated on :11434 (alias qwen3-vl-8b-abliterated), a dense 8B VLM in the same size class as Ornith 1.5 9B. Image in, text out — the third resident model the router can dispatch to.

**How it's built.** A llama-server section in the models.ini preset (explicit mmproj-url, since router-mode --hf-repo does not attach the projector). Q4_K_M, Vulkan, part of the ~10-model catalog under models-max 3.

**Steps in execution.**

1. **Image** — Client sends an image.
2. **Route** — → qwen3-vl-8b via the router.
3. **Text** — Return caption/answer.

### Clients & surfaces

#### D · Hermes Desktop

**In one line.** Hermes Desktop client on talos.

**What it does.** The Hermes Desktop chat surface. Primary is xAI grok-4.6. Vision goes to straylight; approval, titles and web extract go to dixie; subagents go to straylight.

**How it's built.** Desktop app on talos. Vision → straylight :11434 (qwen3-vl-8b). Approval/titles/web_extract → dixie :11434 (ornith-1.5-9b). Delegation → straylight :11434 (ornith-1.5-35b-a3b).

**Steps in execution.**

1. **Turn** — Message from the desktop.
2. **Primary** — xAI grok-4.6.
3. **Aux** — vision → straylight; approval/titles/web_extract → dixie.
4. **Reply** — Streamed back to the surface.

#### P · Local clients

**In one line.** Open WebUI, OpenCode and the Grok TUI — the other surfaces.

**What it does.** Local clients: Open WebUI (:8080), OpenCode (Ornith) and the Grok TUI. Chat flows to llama, speech goes to whisper.

**How it's built.** Each is an OpenAI-compatible client pointed at straylight llama :11434; STT clients hit whisper :11435 first.

**Steps in execution.**

1. **Chat** — → llama :11434.
2. **STT** — → whisper :11435.

#### Cp · chromium-cdp

**In one line.** The browser tools, driven by Chromium over CDP.

**What it does.** chromium-cdp on :9222 powers the Hermes browser tools (page reads, clicks, screenshots). It is a local client of the desktop, not an inference endpoint.

**How it's built.** Chromium with remote debugging on :9222; the browser_* tools drive it. No model involved — it is how the agent sees and clicks the web.

**Steps in execution.**

1. **Connect** — CDP on :9222.
2. **Drive** — Navigate, click, read DOM.
3. **Return** — State back to the agent.

### Memory plane (rift)

#### S · Honcho stack (podman)

**In one line.** The memory plane: Honcho API, deriver, and vector store.

**What it does.** Honcho on rift in podman: the API (deriver), pgvector (:5432) and redis (:6379). Provider is honcho, workspace hermes-fleet.

**How it's built.** podman stack on rift. The deriver has no local gen; every Honcho role (deriver, summary, dream, dialectic, embeddings) calls http://dixie:11434/v1.

**Steps in execution.**

1. **Query** — Deriver receives a recall request.
2. **Call back** — gen/embed/dialectic → dixie :11434.
3. **Store** — pgvector persists, redis caches.

#### Oe · openconcho :8080

**In one line.** The public-facing surface for the Honcho API.

**What it does.** openconcho on :8080, published at https://rift.scylla-hammerhead.ts.net via Tailscale Serve (background, real LE cert). The browser/admin surface for the memory plane.

**How it's built.** Tailscale Serve --bg 8080 → the rift hostname with a real Let's Encrypt cert. Serves openconcho on top of the Honcho API.

**Steps in execution.**

1. **Serve** — openconcho on :8080.
2. **Publish** — Tailscale Serve --bg, LE cert.
3. **Reach** — From any tailnet node.

### Other hosts

#### R · orion — NUC11PAHi5

**In one line.** Non-LLM compute — search and utility services, not inference.

**What it does.** orion, a NUC11PAHi5. No GPU and no llama/Honcho role. It is the fleet's general-purpose compute node for services such as SearXNG (search) and similar utilities.

**How it's built.** NUC11PAHi5, no GPU. Not on the inference path. Runs always-on utility services (Syncthing today; the workload class is search/index/utility rather than models).

**Steps in execution.**

1. **Role** — Non-inference compute.
2. **Services** — Search/utility class (e.g. SearXNG), not LLM serving.

#### Tn · titan — M3 Max MacBook

**In one line.** Strongest GPU on the tailnet — Metal laptop, not always-on infra.

**What it does.** titan, an Apple M3 Max MacBook Pro (16-core CPU / 40-core Metal GPU, 128 GB unified memory, macOS). Hermes gateway plus a local llama.cpp router. Fallback, aux and delegation for titan's own sessions — not a fleet serving host.

**How it's built.** llama.cpp llama-server via launchd, OpenAI-compatible at 127.0.0.1:8081 (also reachable on the tailnet IP), models-max 2. ollama on :11434 is present but not wired into Hermes. macOS sleep means this is not an SLA.

**Steps in execution.**

1. **Power** — M3 Max; 128 GB unified RAM; Metal 3 GPU.
2. **Router** — llama.cpp :8081 (models-max 2); ollama :11434 unwired.
3. **Role** — Own-session fallback/aux/delegation.
4. **Reach** — Laptop — do not put serving or cron here.

#### Dx · dixie — RTX 3060 helper host

**In one line.** Helper-tier CUDA inference — always-on, not the main loop.

**What it does.** dixie, Alienware Aurora R11 (i7-10700, 64 GB, one RTX 3060 12 GB at 140 W). NixOS, headless, Wi-Fi. Honcho compute and small-model aux live here. A second GPU is out, so vision on :11433 cannot load.

**How it's built.** llama.cpp CUDA on the tailnet over HTTP: :11434 router (models-max 3, three residents: ornith-1.5-9b-uncensored, honcho-dialectic, honcho-embed) and :11433 vision (unavailable while the three are resident). Not a main-loop or delegation target (12 GB, two slots per model, Wi-Fi, no TLS).

**Steps in execution.**

1. **Power** — RTX 3060 12 GB @ 140 W; always-on.
2. **Router** — :11434 models-max 3, three residents.
3. **Vision** — :11433 exists but cannot load.
4. **Role** — Honcho + aux. Not TLS. Not high-concurrency.

## Flows (representative packets)

Payload shapes are what the design implies, not measured traffic.

### Talos primary turn (cloud)

| # | From → To | Packet | Representative payload |
|---|---|---|---|
| 1 | T → X | primary request | `{"model":"grok-4.6"}` |
| 2 | X → T | primary response | `{"delta":"Sure —"}` |

### Local aux turn

| # | From → To | Packet | Representative payload |
|---|---|---|---|
| 1 | D → L | subagent/vision | `{"kind":"aux"}` |
| 2 | L → D | response | `{"model":"ornith-1.5-35b-a3b"}` |

### Delegation fallback

| # | From → To | Packet | Representative payload |
|---|---|---|---|
| 1 | T → L | aux/delegation | `{"kind":"delegation"}` |
| 2 | L → T | response | `{"model":"ornith-1.5-35b-a3b"}` |

### Memory recall

| # | From → To | Packet | Representative payload |
|---|---|---|---|
| 1 | G → S | recall | `{"query":"move my 3pm","k":12}` |
| 2 | S → Dx | gen/embed | `{"op":"embed"}` |
| 3 | Dx → S | result | `{"hits":2}` |
| 4 | S → G | memories | `{"hits":2}` |

## Questions — index

Reference by ID. ✓ resolved (with date) · otherwise open.

- **Q-C1** (C) Who reads the claude reply and when does it beat the local router?
- ~~**Q-O1**~~ (O) ✓ Billed per-delegation; kept off the primary path, available on demand (2026-09-21).

## What the platform gives vs what we own

**Platform gives:** Tailscale tailnet (mTLS-ish mesh, one hostname per host), Mattermost (transport + the 'claude' channel), llama-server (OpenAI-compatible router, models-max), podman (Honcho stack).

**We own:** Routing policy in each Hermes gateway (primary vs aux/fallback), the local model catalogs on straylight/titan/dixie, the memory provider config, and the GPU-slot ceilings (Strix Halo models-max 3; titan models-max 2; dixie models-max 3).

## Planned filesystem

```
docs/inference/
  atlas/
    data.mjs          # this file — edit me
    build.mjs         # renderer (from skill)
    template.html     # renderer (from skill)
  atlas.html          # generated (in docs/inference/)
  SYSTEM.md           # generated text twin (in docs/inference/)
  CONTEXT.md          # glossary, by hand
  adr/                # hard-to-reverse decisions (by hand)
  research/           # evidence, append-only
```

## How this file is maintained

Generated from `docs/inference/atlas/data.mjs` by `node docs/inference/atlas/build.mjs`, which also builds the interactive atlas (`atlas.html`). Edit the data file, rebuild, republish — never edit this file by hand.