← Wu Family Server

Wu-House-PC — Admin/Technical Reference

Ignore if you're not Andy (or don't, feel free to read if you're curious). Claude-generated summary of all my notes written throughout the process of setting up the server——for future reference in case I (you) forget how this works.

The machine: Wu-House-PC, Ubuntu 26.04, Ryzen 5 9600X, 24GB Radeon RX 7900 XTX (gfx1100), kernel pinned to 7.0.0-15-generic. Everything below runs on this one box.

---

The big picture

Three services, one GPU:

Two structural facts explain most of the design decisions below:

1. Tailscale is the only front door. No ports are forwarded on the router. Every device that needs access runs the Tailscale client and joins the tailnet, getting a 100.x.x.x address that's meaningful across the whole private group regardless of physical network. Nothing is publicly routable — see "How routers and IP addresses work" for the mechanics of why that matters.

2. There is exactly one GPU, and three things want it. A loaded LLM holds 17–23GB; a game wants the whole card. Rather than trying to share, the system enforces strict turn-taking — covered in full under "GPU arbitration."

---

How a request travels

Client device
   │
   ├─ Tailscale ──────────── encrypted tunnel, or plain LAN if local
   ↓
Wu-House-PC
   │
   ├─ UFW ─────────────────── port + source-range + interface check
   ↓
Target service
   │
   ├─ Open WebUI :3000    ─┐
   ├─ Sunshine :47984-48010 ├─→ GPU
   └─ ttyd :7681 (via Serve)┘

Layers only talk to their immediate neighbor. When something breaks, walk the chain top to bottom — each layer working independently is what makes that tractable.

---

How routers and IP addresses work

Worth having the mental model precise, because every "why won't this connect" question eventually traces back to it.

The server's LAN address, 10.0.0.112, is a private address — RFC 1918 space. Every home network reuses 10.0.0.x / 192.168.x.x; the number is only meaningful inside this one LAN. There is no route from the public internet to it, full stop.

NAT is why. The router holds one public IP from the ISP. Outbound connections get rewritten to that public address on the way out, with a mapping kept so replies find their way back to the right internal device. An unsolicited inbound packet has no mapping to match, so the router drops it — not a policy decision, just a structural consequence of how the translation table is built (outbound-first).

Port forwarding is the traditional workaround — a static rule telling the router "always send port 3000 traffic to 10.0.0.112." Deliberately not used here for Open WebUI: a self-registering login page with a permanent hole through NAT is exactly the kind of thing that gets scanned and probed within hours of being live.

Tailscale sidesteps NAT rather than punching through it. Every device makes an outbound connection to Tailscale's coordination servers first (always permitted by NAT), and that's used to broker a direct connection between two devices — NAT traversal / hole punching. Each device gets a second address, 100.x.x.x, meaningful only within the tailnet, requiring zero router configuration. This is also why moving the whole server to a new house changes nothing for anyone already on the tailnet — the 100.x address isn't tied to the physical network at all.

---

How the firewall works (UFW)

sudo ufw status numbered to view; rules match top-to-bottom, first match wins, default is deny. It's a whitelist.

Anatomy of a rule:

[ 2] 3000    ALLOW IN    10.0.0.0/24

Port, then allowed source range. /24 = 256 addresses (one LAN); /10 = the much larger Tailscale block (100.64.0.0/10). A packet to port 3000 from a Tailscale address doesn't match rule 2 — it matches the separate rule scoped to 100.64.0.0/10. Anything matching neither gets silently dropped; no RST, no response, so a port scan from outside sees nothing at all, not even "closed."

Interface-bound rules are stricter than source-address rules alone:

47989/tcp on docker0    ALLOW IN    172.17.0.0/16

This requires the packet to have physically arrived on the docker0 interface, not just claim a 172.17.x.x source — which a spoofed packet arriving from elsewhere could otherwise fake. Used throughout for Docker-bridge traffic reaching Sunshine.

Design principle to preserve when editing rules: internal plumbing (Ollama's 11434, SearXNG's 8888) is scoped to 172.17.0.0/16 only — reachable exclusively from other containers on the Docker bridge, never from LAN or Tailscale directly. Only Open WebUI, sitting in front with a login, is allowed to reach them. Widening those two ports to LAN/Tailscale would let anyone bypass authentication entirely and hit the raw model API.

Full current rule set and the reasoning behind each one lives in the setup notes' "IP addresses / UFW rules" section — check there before adding anything, since the numbering shifts after any ufw delete.

---

Service 1 — Family LLM

Stack: Open WebUI (Docker, port 3000) → Ollama (systemd service, port 11434) → ROCm → amdgpu driver → GPU.

Open WebUI owns everything user-facing: accounts, chat history (SQLite in the open-webui Docker volume), the model dropdown, per-model permissions, the web-search toggle, and any Filters (see GPU arbitration below). Self-signup is disabled — accounts are created manually via Admin Panel → Users. Because this layer knows nothing about what's underneath, Ollama could be swapped for a different backend without touching a single account or chat.

Config/data locations:

• Container: docker ps, look for open-webui
• Data volume: open-webui → /var/lib/docker/volumes/open-webui/_data/
• Backup: docker run --rm -v open-webui:/data -v $(pwd):/backup alpine tar czf /backup/openwebui-backup.tar.gz -C /data .
• Health check: curl http://localhost:3000/health → {"status":true}

Known port-binding gotcha: this container was at some point recreated bound to the Tailscale IP specifically (100.108.90.123:3000->8080/tcp) rather than 0.0.0.0:3000->8080/tcp. That's why LAN/localhost access can silently stop working while Tailscale access keeps working — check docker ps port column if this happens again. Left as-is by choice (Tailscale-only usage), but it means the container will fail to start if the Tailscale IP ever changes (re-registration, tailnet migration, identity wipe). If that ever happens, recreate with -p 3000:8080 to bind all interfaces, and don't forget the -v /home/sierrawu/.sunshine-state:/sunshine-state:ro mount when doing so (see GPU arbitration — losing this mount silently breaks the gaming guard).

Ollama is the inference engine — stateless, no accounts, just prompt in/tokens out.

• Config override: sudo systemctl edit ollama → writes /etc/systemd/system/ollama.service.d/override.conf
• Current environment: OLLAMA_HOST=0.0.0.0 (required — Docker counts as external), OLLAMA_KEEP_ALIVE=-1 (model never unloads on idle; see GPU arbitration for why this is safe)
• After any edit: sudo systemctl daemon-reload && sudo systemctl restart ollama
• Models registered via Modelfiles in ~/models/Modelfile.*, created with ollama create <name> -f <Modelfile>. Naming convention: Wu-House-LLM:<tag>.
• ollama ps shows what's resident and whether it's on GPU (100% GPU, not CPU) — the single most useful diagnostic command on this whole box.

ROCm is the only reason Ollama gets ~20-25 tok/s instead of ~2. rocminfo | grep gfx must show gfx1100; rocm-smi for live status. Required group membership: render, video — check with groups (needs a fresh login after any change, doesn't apply live).

Web search: the model never queries anything itself. Open WebUI hits SearXNG (~/searxng/settings.yml, Docker container on port 8888, must have formats: [html, json] or the integration silently breaks) and pastes results into the prompt before the model ever sees the request. This means search behaves identically regardless of which model is selected — it's a server-side property, not a model capability.

One-model-at-a-time: 24GB VRAM fits one large model. Switching in the dropdown means unload + reload (20–30s). The family-facing dropdown intentionally exposes one general model; coding/reasoning models are admin-only via Admin Panel → Settings → Models, to avoid accidental thrash.

---

Service 2 — Remote gaming and desktop

Sunshine (server-side capture + encode + stream) pairs exclusively with Moonlight (client). Config: https://localhost:47990 (LAN-only by design — deliberately not opened to Tailscale, tunnel via SSH if remote admin access is ever needed: ssh -L 47990:localhost:47990 sierrawu@100.108.90.123). Encoder must be VAAPI (AMD's Linux hardware encoder — AMF does not exist on Linux, ignore guides that say otherwise), capture method KMS (the only Wayland-compatible path on this box, since there's no Xorg fallback on 26.04).

Mandatory KMS capture permission, wiped by every Sunshine package update:

sudo setcap cap_sys_admin+p $(readlink -f $(which sunshine))

Verify with getcap $(readlink -f $(which sunshine)) — should show cap_sys_admin+p. Empty output after an update is the #1 cause of "gaming just stopped working, black screen, no error."

Native Moonlight

LAN auto-discovery, or manual entry via Tailscale IP (100.108.90.123) from off-network. Pairs once via 4-digit PIN entered in Sunshine's web UI. Moonlight stores multiple addresses per host (LAN, public IP, IPv6, Tailscale) and falls back automatically — see a paired device's "connection details" screen for the full set.

Browser Moonlight (Moonlight Web)

Container moonlight-web, image mrcreativ3001/moonlight-web-stream:latest, listening on 127.0.0.1:8080 internally, exposed via Tailscale Serve (not a raw Tailscale IP — a separate HTTPS reverse-proxy layer):

sudo tailscale serve --bg --https=443 8080

Container → Sunshine hop goes over host.docker.internal (Docker's internal bridge), not LAN or Tailscale at all. Default user is a shared non-admin Family account, set via default_user_id in /moonlight-web/server/config.json inside the container. Direct desktop links (?hostId=…&appId=…) stay stable only as long as the moonlight-server volume and Sunshine's Desktop app entry aren't deleted/recreated.

Tailscale Serve mappings currently in use (port+path keyed — setting a new mapping on the same port/path silently evicts the old one, which is how Beszel once knocked moonlight-web off the root path):

--https=443 8080     → moonlight-web (root)
--https=8443 7681    → thermals dashboard

tailscale serve status to inspect; Serve is tailnet-only (Funnel would be public, deliberately unused).

Steam

Proton enabled (Settings → Compatibility). protondb.com before trusting any given game; kernel-level anti-cheat titles (Valorant, Fortnite, Destiny 2) will never work here. One Steam session per install — family members log in/out through the real, streamed desktop, so credentials never transit the server.

---

Service 3 — Thermals dashboard

temps script (/usr/local/bin/temps, originally ~/temps.sh) wrapped by ttyd, a read-only terminal-to-browser bridge:

/etc/systemd/system/temps-web.service
ExecStart=/usr/bin/ttyd -p 7681 -i 127.0.0.1 -t fontSize=15 -t disableLeaveAlert=true /usr/local/bin/temps

Security notes, don't undo these: never add -W/--writable — this is meant to be display-only. Never run ttyd with no command argument — that spawns a login shell, handing anyone on the tailnet a root-adjacent terminal as sierrawu. -i 127.0.0.1 binds loopback-only so Tailscale Serve is the only path in.

GPU junction temp is the number that matters: healthy under ~95°C, junction–edge delta under ~20°C. A widening delta over months usually means thermal paste/pad degradation, not sensor drift.

---

GPU arbitration — how three services share one card

This is the part most likely to silently break after an unrelated change, so it gets the full technical writeup.

The policy

A model stays resident in VRAM indefinitely (OLLAMA_KEEP_ALIVE=-1 — see Service 1). The only thing that evicts it is a gaming session starting. First prompt after gaming pays a 20–30s reload; every prompt before or after that is instant.

The two independent mechanisms

1. Request-time block. An Open WebUI Filter ("Gaming GPU Guard", Admin Panel → Functions — must show Active and Global, both toggles, easy to miss one) checks for a marker file on every inbound chat request:

if os.path.exists("/sunshine-state/gaming"):
    raise Exception("...")   # stops before Ollama is ever called

That path exists inside the container only because of a mount added when Open WebUI was last recreated:

-v /home/sierrawu/.sunshine-state:/sunshine-state:ro

Losing this mount on a future container recreation is the single easiest way to silently break gaming protection — the filter will run, find nothing at that path, and simply never block. Verify after any Open WebUI recreation: docker exec open-webui sh -lc 'test -e /sunshine-state/gaming && echo GAMING || echo FREE'.

2. VRAM eviction. moonlight-status.sh (already responsible for the gaming marker itself, watching UDP 47998 for Sunshine video traffic) also evicts any resident model on every polling pass while a stream is active — not just once at stream start:

loaded=$("$OLLAMA" ps | awk 'NR>1 && $0 !~ /Stopping/ {print $1}')
if [ -n "$loaded" ]; then
    logger -t moonlight-status "unloading $loaded"
    echo "$loaded" | xargs -r -n1 "$OLLAMA" stop
fi

Why recurring and not one-shot at stream start: the Filter only covers requests routed through Open WebUI's normal chat path. Background tasks (title/tag generation, follow-up suggestions), model warmup on dropdown selection, or anything hitting the Ollama API directly all bypass it. Under -1 keep-alive, a stray load from any of those would never expire on its own — observed once in testing, a model reloaded 62 seconds into an active gaming session and was caught within 2 seconds by the recurring check. Edge-triggered-only would have left it resident for the rest of the session.

Why gaming wins rather than the AI: an earlier design killed the running game 10 seconds after an AI prompt arrived. Deliberately removed — destroying unsaved progress to serve a chat message is the wrong tradeoff. Current behavior shows the requester a clear message and leaves the decision to a human.

Three operational gotchas, each cost real debugging time and will recur if not respected when editing this script

$HOME must be set explicitly in the service unit. The Ollama CLI panics on startup without it (panic: $HOME is not defined), and systemd units don't provide it by default. Symptom is maximally deceptive: systemctl status shows active (running), but ollama ps inside the script silently returns nothing, so the eviction logic no-ops every single pass with no error anywhere in the normal log stream. Fix, already applied:

# moonlight-status.service override
[Service]
Environment="HOME=/root"

If this script is ever rewritten from scratch, this line is not optional.

logger tags by invoking user, not by string content. logger "moonlight-status: text" does NOT make the journal entry queryable via journalctl -t moonlight-status — running as root, it tags as root, burying every line among unrelated root activity. Must be logger -t moonlight-status "text" explicitly. (The old, now-disabled llm-priority.sh has this same bug if it's ever revived — don't trust its journal output either.)

Ollama's Stopping... state isn't instant, and the eviction loop must exclude it. A 23GB model can take up to ~60s to actually unload. Without filtering rows in that state out of the ollama ps parse, the loop re-issues stop against an already-terminating process every ~2 seconds — harmless functionally, but produces 20+ duplicate log lines per single eviction and is worth keeping the !~ /Stopping/ guard for signal clarity.

Failure mode introduced by this design

Under -1 keep-alive, the watcher service is now the only thing that ever frees VRAM. Previously (5m keep-alive) a stuck flag or dead watcher self-corrected within minutes. Now, if moonlight-status.service dies, a model holds the card indefinitely and gaming will be starved with no automatic recovery. First check if gaming is ever inexplicably poor: sudo systemctl status moonlight-status.service. Fallback if this class of problem recurs: revert OLLAMA_KEEP_ALIVE to 5m — the original edge case (gaming within 5 minutes of a chat prompt) is far cheaper than a permanently stuck model with a dead watchdog.

---

How the server stays online

Assumption baked into every layer below: nobody is physically present.

1. BIOS — survives a power outage

ASUS TUF Gaming B650E-E WiFi → Advanced Mode (F7) → Advanced → APM Configuration → Restore AC Power Loss → Power On (not Last State — ambiguous if the crash failsafe fires mid-outage; not Power Off — leaves it dark until a human intervenes).

This only handles recovery after power returns — the shutdown itself is still hard and uncontrolled. A UPS is the missing piece for that half, not currently in place.

2. Auto-login — required for the desktop session to exist

sierrawu auto-login is enabled (/etc/gdm3/custom.conf, AutomaticLoginEnable=true). Sunshine streams a real graphical session; without one existing, gaming is dead until someone physically logs in, auto-login or not. Verify: grep AutomaticLogin /etc/gdm3/custom.conf.

3. Everything autostarts

Check enablement in bulk: systemctl is-enabled ollama tailscaled docker watchdog, docker inspect open-webui --format '{{.HostConfig.RestartPolicy.Name}}'.

4. Two-layer crash failsafe

Root cause is an unfixed amdgpu/Wayland kernel regression (ring gfx_0.0.0 timeout with gnome-shell attached, failed GPU recovery). Confirmed independent of Ollama — reproduces with zero inference load. Current mitigation is pinning kernel 7.0.0-15-generic (sudo apt-mark hold linux-image-7.0.0-15-generic linux-image-generic — note this only blocks upgrades, it does NOT control what GRUB actually boots; always verify with uname -r after any kernel-touching apt upgrade, and set GRUB_DEFAULT explicitly in /etc/default/grub if it ever drifts).

Layer 1 — hardware watchdog. SP5100/SB800 chipset timer, OS-independent. watchdog.service pets it every few seconds; 60s of silence triggers a hardware-level force reboot, no OS cooperation needed.

• Config: /etc/watchdog.conf (watchdog-timeout = 60, max-load-1 = 24)
• Known gotcha, already fixed once, will recur on kernel changes: Ubuntu ships a per-kernel modprobe blacklist including sp5100_tco. When active, the daemon reports active/running while /dev/watchdog never actually opens (cannot open /dev/watchdog (errno=2)) — completely silent until a real crash goes uncaught. Check: grep sp5100_tco /usr/lib/modprobe.d/blacklist_linux_*.conf — every line should read #blacklist sp5100_tco (commented). If any kernel installs a fresh blacklist file, this needs reapplying and verifying on a real cold boot, not a service restart (lsmod | grep sp5100, sudo journalctl -u watchdog -b -n 15 should show hardware watchdog identity: SP5100 TCO timer, never cannot open /dev/watchdog).

Layer 2 — GPU health check. /usr/local/bin/gpu-healthcheck.sh, root cron every 5 minutes, timeout 10 rocm-smi — hang or failure triggers /sbin/reboot. Exists separately from Layer 1 because observed crashes sometimes killed only the GPU while the OS stayed technically responsive (SSH laggy but alive) — a narrower failure Layer 1's whole-system check wouldn't catch. Log trail: sudo grep -a gpu-healthcheck /var/log/syslog (-a required — syslog binary content otherwise makes plain grep report "binary file matches" and hide results).

Standing lesson, now confirmed twice (once with the watchdog, once with the moonlight-status eviction script): a service reporting active (running) is not evidence it's doing its job. Check what it's actually printing — journalctl, not just systemctl status's top line — and verify any fix on a real cold boot, since module/state tricks can look correct on a live system while still failing at actual boot time.

If a crash ever goes uncaught past 5-6 minutes: don't keep waiting on the failsafe, hold power 10s and cycle manually.

---

Dependency map — what breaks what

The failures that don't announce themselves. Check this before making a change in any of these areas.

---

Cast of services — quick definitions

---

Quick reference

New family member needs: Tailscale invite (single-use), Open WebUI account (Admin Panel → Users → Add User), and — for native gaming — the Moonlight app paired via PIN.

Server relocation: nothing changes for end users — Tailscale addresses, bookmarks, accounts, and pairings all survive. Only the six LAN-scoped UFW rules need the new subnet.

---

Known gaps

• No UPS. BIOS recovery is after power returns; the shutdown itself is still hard and uncontrolled.
• The amdgpu/Wayland driver bug is unfixed, only mitigated via kernel pin. If crash frequency ever climbs to multiple times/day, that's the signal to actually chase it (newer kernel, amdgpu.gfxoff=0, or a distro reinstall) rather than continuing to lean on the failsafes.