ws: migrate push stream to data streaming

- daemon: send full snapshot on connect; versions/tick now carry the
  full state of one subsystem (subsystem + data); no legacy
  updated/subsystems payloads; refresh_state and POST /status/refresh
  broadcast per-subsystem versions with data
- client: modelSet() patches models in place; onMessage/topic refresh
  retired; 3s initial-load fallback via new POST /api/status/refresh
- schema: lib/schema.py TypedDicts + hoover/schema.js defaults +
  docs/state-model.md as single source of truth for state shapes
- system: poll at 1s, volatile metrics registered, dashboard uses a
  dedicated system model (status model removed)
- firewall: refuse to strip both https and ssh from the default zone
  (409, force override via UI confirm); set_zone_services persists
  services to the declarative config; collector exposes default_zone
- UI: pages migrate to flat state shapes; post-mutation modelFetch
  refreshes removed (WS delta covers it)
- tests: ws snapshot/delta/broadcast, refresh-state, schema types,
  model-set/js ws handler and reconnect fallback
This commit is contained in:
2026-08-20 01:38:00 +00:00
parent 9c9f92ad04
commit 332d14e37d
45 changed files with 2819 additions and 496 deletions
+49 -7
View File
@@ -1947,6 +1947,40 @@ Apply pending changes for all subsystems in dependency order.
| `applied` | `[string, ...]` | List of subsystems that were applied |
| `errors` | `[object, ...]` | Any errors encountered during apply |
---
#### Refresh State
```
POST /api/status/refresh
```
Re-collect state from the daemon, optionally filtered by subsystem. Proxies the daemon's `POST /status/refresh`, which populates the state store for the requested subsystems, replies with their current state, and broadcasts a `versions` WS delta for each so all connected viewers stay in sync.
**Request Body** (optional — `{}` or omitted refreshes all subsystems):
| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `subsystems` | `[string, ...]` | No | Subsystem names to refresh (e.g., `["firewall"]`) |
**Response (`data`):**
| Field | Type | Description |
|-------|------|-------------|
| — | `object` | Map of the requested subsystem name(s) to its full state dict (`null` = collector not populated / failed) |
**Example:**
```json
// Request
{"subsystems": ["firewall"]}
// Response
{"ok": true, "data": {"firewall": {"config": {...}, "zones": {...}, "active_zones": {...}, "timestamp": "..."}}}
```
Returns HTTP `500` if the daemon is unreachable.
### Sysctl
#### Set Kernel Parameter
@@ -2069,14 +2103,22 @@ Returns HTTP `404` if the log file does not exist.
## WebSocket Protocol
The daemon exposes a WebSocket at `/ws` (port 9091) for real-time state change notifications. On connect, the server sends:
The daemon exposes a WebSocket at `/ws` (port 9091) for real-time state streaming. After authentication, the server pushes a full state snapshot on connect and then per-subsystem deltas — the client patches models in place (`modelSet`) with no HTTP round-trip.
```json
{"type": "init", "versions": {"firewall": 0, "dnsmasq": 0, ...}}
```
### Handshake Authentication
The JWT **access** token travels as the **raw `Sec-WebSocket-Protocol` subprotocol name** (the bundled client sends the bare token, no `Bearer ` prefix — subprotocol names must be valid RFC 6455 tokens). The daemon additionally accepts a legacy `Bearer <token>` subprotocol (non-browser clients) and an `X-Auth-Token` header fallback. The token is validated without session binding (browsers cannot send custom headers on the WebSocket handshake) but with the jti revocation check. A missing or invalid token yields HTTP `401` and no socket is opened.
### Message Types
- **`versions`** — Structural state change. `updated` contains subsystem names whose version counters changed. Triggers full re-fetch.
- **`tick`** — Volatile-only change (stats, counters, DHCP IPs). `subsystems` contains affected subsystem names. Triggers lightweight per-subsystem re-fetch.
- **`notify`** — Single-topic notification. `topic` is the subsystem name.
| Type | Sent | Fields | Meaning |
|------|------|--------|---------|
| `snapshot` | On connect (after auth) | `data: {subsystem: state\|null, …}` | Full state for every subsystem. `null` = collector not populated / failed — clients skip those entries. |
| `versions` | Structural change | `subsystem`, `data` | The full state of the one changed subsystem (zone added, config changed, …). Version counter bumped; data pushed. |
| `tick` | Volatile-only change | `subsystem`, `data` | The full state of the one changed subsystem (stats/counters/DHCP IPs). No version bump. |
There is no legacy `updated` dict or `subsystems` array — each data-carrying message names a single `subsystem` and carries its full `data`.
### Manual Refresh
`POST /api/status/refresh` re-collects state (optionally filtered by a `subsystems` array) and broadcasts a `versions` delta for each requested subsystem. It is the HTTP fallback the client uses for the initial load (3s timer) and reconnect recovery. See the [Status API — Refresh State](#refresh-state) section for the full request/response contract.
+12 -9
View File
@@ -148,19 +148,22 @@ The daemon runs background polling tasks for subsystems with external runtime st
| wireguard | 10s | Peer connections/handshakes change frequently |
| dnsmasq | 10s | Lease file + service status |
| networkd | 10s | Interface up/down, DHCP address changes |
| system | 1s | Real-time metrics (load/memory/swap/traffic) |
| nginx | 60s | Config-file drift self-heal (lazy in-place migration) |
| acme | 300s | Config-file drift self-heal (lazy in-place migration) |
nginx, acme, and auth are not polled — they have no external runtime state.
Only `auth` is not polled — it has no external runtime state.
**Two-layer diff:** Each poll cycle classifies changes as:
- **Structural change** (zones added, peers removed, config changed): triggers `bump()` + broadcast `{"type": "versions", ...}` → full UI re-load
- **Volatile change only** (transfer counters, DHCP-assigned IPs): sends `{"type": "tick", "subsystems": [...]}` → lightweight per-subsystem re-fetch
- **Structural change** (zones added, peers removed, config changed): triggers `bump()` + broadcast `{"type": "versions", "subsystem": ..., "data": ...}` → daemon pushes the full subsystem data over WS; the client patches the model in place via `modelSet`
- **Volatile change only** (transfer counters, DHCP-assigned IPs): sends `{"type": "tick", "subsystem": ..., "data": ...}` → same in-place patch, without a version bump
- **No change**: silence
Volatile fields per subsystem: `wireguard` (peer transfer/handshake stats), `firewall` (DHCP-assigned IPs), `networkd` (DHCP addresses, link metrics). Defined per collector via `register_volatile()`.
Volatile fields per subsystem: `system` (load/memory/swap/traffic), `wireguard` (peer transfer/handshake stats), `firewall` (DHCP-assigned IPs), `networkd` (DHCP addresses, link metrics). Defined per collector via `register_volatile()`.
Poll intervals are configurable via `VACUUM_WALL_POLL_INTERVALS` env var (`firewall:30,wireguard:10,...`).
On collector failure during a poll, no broadcast is sent (avoids noisy ticks). State data is set to `None`.
On collector failure during a poll, no broadcast is sent (avoids noisy ticks). State data is set to `None`, and `broadcast_versions` additionally skips a `None` payload defensively (a null payload would clobber good client data — the next successful poll or mutation broadcasts the real value).
## System Config Import
@@ -338,11 +341,11 @@ Client loads /static/app.js ──→ Hoover initializes, checkSession() (401 wi
Authenticated ──→ mounts #sidebar and #main render roots
apiFetch() ──→ injects Authorization: Bearer <token> header ──→ Flask REST API
Flask before_request ──→ validates JWT from header, checks blacklist, verifies permissions
Hoover connects WebSocket ──→ daemon/ws (127.0.0.1:9091?token=<access_token>)
Hoover connects WebSocket ──→ daemon/ws (raw JWT as Sec-WebSocket-Protocol subprotocol name; legacy `Bearer <token>` subprotocol + X-Auth-Token header fallbacks accepted)
Page navigate (hash change) ──→ reactive router state updates ──→ render engine re-executes ──→ VDOM diff patches DOM
User action (form submit) ──→ apiFetch() ──→ Flask REST API ──→ daemon/client.py ──→ vacuum-walld
Token expiry ──→ refreshScheduler() ──→ POST /api/auth/refresh ──→ new tokens
WebSocket message (versions) ──→ topic match ──→ page load() re-executed ──→ state updated ──→ render engine patches DOM
WS connect ──→ snapshot (full state) / versions + tick deltas (per-subsystem data) ──→ modelSet() patches model in place ──→ render engine VDOM-diffs and patches only changed DOM nodes
```
The SPA entry point only serves `index.html` at `/`. All other paths return 404. Non-API, non-static paths are not served by Flask — the client-side router handles all navigation via hash changes. A dedicated `/vendor/<path>` route serves vendored JS libraries.
@@ -355,9 +358,9 @@ Each route is a `definePage()` component with reactive state, async data loading
All JavaScript is served as ES modules. Cache invalidation is handled via HTTP cache-control headers. Dev mode (`VACUUM_WALL_DEV`) disables aggressive static asset caching.
### WebSocket Broadcast
### WebSocket Data Streaming
The daemon broadcasts state-change notifications via WebSocket. Hoover's `subscribe` mechanism maps page-level topic subscriptions to automatic `load()` re-executions. Messages are debounced (300ms) and in-flight loads are aborted before re-loading, ensuring the UI always displays the latest available data.
The daemon pushes state over the WebSocket — no HTTP round-trip for auto-refresh. On connect, after the JWT handshake, it sends a full snapshot (`{"type": "snapshot", "data": {subsystem: state|null, …}}`). On every structural change it broadcasts a per-subsystem delta (`{"type": "versions", "subsystem": …, "data": …}`); on volatile-only changes it sends `{"type": "tick", "subsystem": …, "data": …}`. The client's `handleMessage` patches the matching reactive model in place via `modelSet()`, and the VDOM diff touches only the changed nodes. HTTP remains the fallback for the initial load (3s timer) and for reconnect recovery.
## Zone Model
+4
View File
@@ -534,6 +534,10 @@ The `zones` object maps zone names (keys) to zone configurations. Each zone corr
The `config_pending()` function compares the declarative config in `config/firewall/config.json` against the live firewalld state returned by the daemon (via `daemon.handlers.firewall.get_state()`). It returns a diff indicating which zones have pending changes for interfaces, services, target, masquerade, forward ports, and rich rules. Zones that exist live but not in config are reported as `unmanaged_zones`.
Both `/api/firewall/zones/<name>/services` and `/api/firewall/config/apply` reconcile **remove-then-add** against the live zone, so anything opened outside the declarative config (e.g. directly via `firewall-cmd`) is reverted on the next apply. Service changes made through the API are persisted to `config.json` to prevent this drift.
**Management-lockout guard.** The firewalld *default zone* is the catch-all for interfaces with no explicit assignment (typically the WAN), and it carries the management plane (nginx https) plus remote recovery (ssh). Changing the default zone's service set so that **neither `https` nor `ssh`** remains raises `409 Conflict` — from `POST /firewall/zones/<name>/services` and `POST /firewall/config/apply` — before any mutation runs. Send `"force": true` in the request body to override (the UI shows a confirm dialog with this effect on the Zones page). If the default zone cannot be determined, the guard fails closed.
## Networkd (IP Configuration)
**File**: `config/network/config.json`
+2 -2
View File
@@ -176,8 +176,8 @@ Log in with the username and password you provided during installation.
1. Confirm `config/auth/config.json` exists with JWT secret and WebAuthn RP configuration
2. Confirm `data/auth.db` exists with admin user present
3. Nginx config no longer has `auth_basic` for management domain
4. WebSocket location no longer has `auth_basic off`
3. Confirm the management server block has no server-level `auth_basic` directive — the management UI is authenticated by the Flask-layer JWT middleware, not nginx
4. Confirm `location /ws` has `auth_basic off` — the WebSocket is authenticated by the daemon via the raw-JWT `Sec-WebSocket-Protocol` subprotocol, never by nginx
5. Access the WebUI at `https://<management-domain>` — should show a login page
### Certificate Note
+154 -91
View File
@@ -11,9 +11,9 @@ Hoover is the custom reactive SPA framework used by the Vacuum Wall web UI. It p
| Render | `render.js` | Render engine: container-level diffing, component lifecycle |
| Component | `component.js` | Page definitions, lifecycle hooks, state caching |
| Router | `router.js` | Hash-based SPA router, `Link` navigation component |
| Model | `model.js` | **Central** reactive store per subsystem, fetch, WS invalidation, loading states |
| Model | `model.js` | **Central** reactive store per subsystem: WS streaming in (`modelSet`), HTTP fallback fetch (`modelFetch`), loading states |
| Auth model | `auth_model.js` | Token/session lifecycle model: storage, refresh scheduling, session validation, login/logout transitions |
| WebSocket | `websocket.js` | Auto-reconnect WS, topic routing to model refresh, `disconnect()` (terminal-auth socket teardown) |
| WebSocket | `websocket.js` | Auto-reconnect WS: streams state to models (`snapshot` on connect → `modelSet`; per-subsystem `versions`/`tick` deltas → `modelSet`), `disconnect()` (terminal-auth socket teardown) |
| API | `api.js` | JSON fetch wrapper, toast notifications, form submissions |
| Helpers | `helpers.js` | Escaping, DOM value helpers, zone parsing |
| Components | `components/*.js` | Reusable UI: layout, data tables, modals, toasts |
@@ -26,12 +26,12 @@ All public APIs are exported from `hoover/index.js`. Pages and app bootstrap imp
```
index.html — static shell with #sidebar, #main, #modal-root
└── app.js — SPA bootstrap
├── modelRegister('firewall', { subsystem: 'firewall', fetch: ... })
├── modelRegister('dnsmasq', { subsystem: 'dnsmasq', fetch: ... })
├── modelFetch('firewall') / modelFetch('dnsmasq') / ...
├── render(sidebarEl, Sidebar) — sidebar render root
├── render(mainEl, MainContent) — main content render root
└── connect() — WebSocket lifecycle
├── modelRegister('firewall', { subsystem: 'firewall', fetch: ... })
├── modelRegister('dnsmasq', { subsystem: 'dnsmasq', fetch: ... })
├── fetchInitialData() — 3s WS-snapshot fallback + non-state fetches
├── render(sidebarEl, Sidebar) — sidebar render root
├── render(mainEl, MainContent) — main content render root
└── connect() — WebSocket lifecycle (snapshot → modelSet)
```
The HTML shell (`index.html`) provides named DOM containers (`#sidebar`, `#main`) plus a `#modal-root` anchor for modals. The `app.js` bootstrap mounts Hoover render functions onto `#sidebar` and `#main`, creating two independent render roots.
@@ -41,14 +41,16 @@ Each render root registers a render function via `render(container, fn)`. When r
### Data Flow
```
WS message → refreshByTopic(topic) → modelFetch(name) → model.data = apiFetch()
→ reactivity proxy triggers render
→ page.render(state) reads model data
WS message → modelSet(name, data) → model.data (reactive proxy) → page.render(state) reads model data
(snapshot on connect, versions/tick deltas per subsystem)
HTTP fallback (initial load 3s timer, reconnect recovery) → modelFetch(name) → model.data = apiFetch()
```
The **model layer** is the single source of truth for subsystem data. Pages never call `apiFetch` for data loading — they call `getModel(name)` in `init()` to get a reactive model, then read `model.data`, `model.loading`, and `model.error` in `render()`.
Mutations (`ConfirmDelete`, `ActionButton`, `QuickModal`, `apiSubmit`) refresh models by name (`refresh: 'firewall'`), not by calling load functions. The model layer ensures in-flight dedup, loading flag management, and WS-driven auto-refresh.
State-backed models receive their data primarily over the WebSocket: the daemon sends a full **snapshot** on connect and per-subsystem **deltas** (`versions` for structural changes, `tick` for volatile-only changes). `handleMessage` patches the matching model in place via `modelSet()` — no HTTP round-trip for auto-refresh. `modelFetch` remains only as the HTTP fallback (a 3-second timer kicks in if the snapshot hasn't arrived) and for the few non-state models (`backends`, `logs`).
Mutations no longer trigger explicit model refreshes: after a successful write the daemon re-collects the affected subsystems and broadcasts WS deltas, which `modelSet` applies. `ConfirmDelete` / `ActionButton` / `apiSubmit` therefore skip `modelFetch` (the legacy `refresh` prop is accepted but ignored). Non-state models that still need a post-mutation fetch wire it explicitly (e.g. `backends` via `onComplete` / `onSuccess`).
## Bootstrap
@@ -58,21 +60,52 @@ The app starts from `webui/static/app.js`:
import { h, render, Link, hComp, ToastContainer, connect, apiFetch,
modelRegister, modelFetch, reactive } from '/static/hoover/index.js';
// 1. Register subsystem models
modelRegister('firewall', {
subsystem: 'firewall',
fetch: async () => {
const r = await apiFetch('/api/firewall/config');
if (!r.ok) throw new Error(r.error);
return r.data;
},
});
// 1. Register subsystem models. All state-backed models share the same
// HTTP-fallback fetch (POST /api/status/refresh, subsystem filter); the
// primary data path is the WS snapshot + deltas (modelSet).
const STATE_MODELS = [
{ name: 'firewall', subsystem: 'firewall' },
{ name: 'dnsmasq', subsystem: 'dnsmasq' },
{ name: 'nginx', subsystem: 'nginx' },
{ name: 'acme', subsystem: 'acme' },
{ name: 'wireguard', subsystem: 'wireguard' },
{ name: 'network', subsystem: 'networkd' },
{ name: 'system', subsystem: 'system' },
];
for (const { name, subsystem } of STATE_MODELS) {
modelRegister(name, {
subsystem,
defaultData: SUBSYSTEMS[subsystem].defaults,
fetch: async () => {
const r = await apiFetch('/api/status/refresh', {
method: 'POST',
body: { subsystems: [subsystem] },
});
if (!r.ok) throw new Error(r.error);
const payload = r.data?.[subsystem];
if (payload == null) throw new Error(subsystem + ': state not populated yet');
return payload;
},
});
}
modelRegister('backends', { subsystem: 'nginx', fetch: async () => { /* /api/proxy/backends */ } });
modelRegister('logs', { subsystem: '*', fetch: async (signal, tab) => { /* LOG_TABS[tab] */ } });
```javascript
// ... more modelRegister calls ...
// 2. Initial fetch for all models
for (const name of ['status', 'firewall', 'network', 'dnsmasq', 'nginx', 'acme', 'wireguard']) {
modelFetch(name);
// 2. Initial data. State-backed models receive their first data via the WS
// snapshot; a 3s timer falls back to modelFetch (HTTP) if it hasn't arrived.
// Non-state models fetch immediately.
function fetchInitialData() {
for (const { name } of STATE_MODELS) {
setTimeout(() => {
const model = getModel(name);
if (model.loading) modelFetch(name); // snapshot not yet delivered
}, 3000);
}
modelFetch('backends');
modelFetch('logs', 'journal');
}
// 3. Create reactive router state
@@ -130,33 +163,39 @@ Manually schedule a re-render. Only one microtask is queued regardless of how ma
## Model
The model layer (`model.js`) is the **central** data synchronization mechanism. Each subsystem gets one reactive model with `{ data, loading, refreshing, error }`. Hoover handles fetching, WS invalidation, loading states, and in-flight dedup.
The model layer (`model.js`) is the **central** data synchronization mechanism. Each subsystem gets one reactive model with `{ data, loading, refreshing, error }`. Hoover handles WS streaming (via `modelSet`), HTTP fetching (fallback + non-state models, via `modelFetch`), loading states, and in-flight dedup.
### `modelRegister(name, definition)`
Register a subsystem model at app bootstrap.
```javascript
// State-backed model — the fetch below is the HTTP *fallback* (POST
// /api/status/refresh with a subsystem filter); the primary path is the WS
// snapshot + per-subsystem deltas applied via modelSet().
modelRegister('firewall', {
subsystem: 'firewall', // WS topic to listen for ('*' = all)
fetch: async (signal) => { // async fetch function
const r = await apiFetch('/api/firewall/config', { signal });
subsystem: 'firewall', // daemon subsystem ('*' = all)
defaultData: SUBSYSTEMS['firewall'].defaults, // schema defaults until first data
fetch: async (signal) => { // HTTP fallback
const r = await apiFetch('/api/status/refresh', {
method: 'POST',
body: { subsystems: ['firewall'] },
});
if (!r.ok) throw new Error(r.error);
return r.data;
return r.data?.firewall; // null → throw so stale data is kept
},
defaultData: null, // optional, initial data value
// onSuccess: (name, data, param?) => { }, // optional — after model.data is set (also for null)
// onFailure: (name, error) => { }, // optional — after model.error is set (real throws only)
});
// Parameterized example — tab-aware fetch:
// Parameterized example — tab-aware fetch (non-state model):
modelRegister('logs', {
subsystem: '*',
fetch: async (signal, tab) => {
const url = LOG_TABS[tab || 'journal'];
const r = await apiFetch(url, { signal });
if (!r.ok) throw new Error(r.error);
return (r.data || '').split('\n').filter(l => l.length > 0);
return { data: (r.data || '').split('\n').filter(l => l.length > 0), tab: tab || 'journal' };
},
});
```
@@ -164,7 +203,7 @@ modelRegister('logs', {
| Parameter | Description |
|---|---|
| `name` | Model name (e.g., `'firewall'`, `'dnsmasq'`) |
| `definition.subsystem` | WS topic string. Use `'firewall'`, `'dnsmasq'`, etc. Use `'*'` to match all topics. |
| `definition.subsystem` | The daemon subsystem this model maps to (`'firewall'`, `'dnsmasq'`, `'networkd'`, …). Used by `refreshByTopic()` for manual / non-WS refresh; `'*'` matches all topics. (The WS stream in `websocket.js` resolves subsystem → model via its own internal map, so `networkd` correctly lands on the `network` model regardless of this field.) |
| `definition.fetch(signal?, param?)` | Async function that fetches and returns data. Throws on error. Receives optional `AbortSignal` and optional parameter (e.g., tab key). |
| `definition.defaultData` | Optional initial data value (default: `null`) |
| `definition.onSuccess(name, data, param?)` | Optional lifecycle hook called after `model.data` is assigned — including `data === null` (a resolved `null` is normal, not an error). `param` is the action object passed to `fetch` (or `undefined`), so hooks can tell which action produced the data. Fire-and-forget: hook errors are caught and logged via `console.warn`; they never clobber `model.error`, the returned promise, or the `finally` flag clearing. |
@@ -172,7 +211,7 @@ modelRegister('logs', {
### `getModel(name)`
Get a reactive model by name. Returns the model object with `{ data, loading, refresh, error }` properties. Call in `init()` to access model state in `render()`.
Get a reactive model by name. Throws if not registered. Returns the model object with `{ data, loading, refreshing, error }` properties. Call in `init()` to access model state in `render()`.
```javascript
// In page init
@@ -197,18 +236,24 @@ render(state) {
Trigger a fetch for the named model. In-flight dedup ensures concurrent callers get the same promise. Updates `loading`/`refreshing` flags automatically.
```javascript
// Initial load
// HTTP fallback for a state-backed model (WS snapshot is the primary path;
// app.js kicks in with modelFetch(name) if no snapshot arrives within 3s)
modelFetch('firewall');
// Post-mutation refresh
const r = await apiFetch('/api/firewall/zones', { method: 'POST', body });
if (r.ok) modelFetch('firewall');
// Parameterized fetch (e.g., tab-aware logs)
// Non-state models fetch directly (not backed by the daemon state store)
modelFetch('backends');
modelFetch('logs', 'journal');
modelFetch('logs', 'nginx-access');
```
> **State-backed models** (`firewall`, `dnsmasq`, `nginx`, `acme`, `wireguard`,
> `network`, `system`) receive their data over the WebSocket snapshot + per-subsystem
> deltas — `modelSet` applies it in place with no HTTP round-trip. After a mutation the
> pages **do not** call `modelFetch`; the daemon re-collects the affected subsystems and
> broadcasts a delta that `modelSet` applies. `modelFetch` for a state-backed model is
> only the explicit / fallback path (its `fetch` hits `POST /api/status/refresh` with a
> subsystem filter). Non-state models (`backends`, `logs`) always fetch via `modelFetch`.
**Behavior:**
- If a fetch is already in progress for this model (and param), returns the existing promise (dedup).
- Sets `model.loading = true` on first fetch, `model.refreshing = true` on subsequent fetches.
@@ -219,9 +264,32 @@ modelFetch('logs', 'nginx-access');
- Does not abort in-progress fetches — other consumers may still need the data.
- The `param` argument is passed to `fetch(signal, param)` for parameterized models. Dedup key is `name` (no param) or `name: JSON.stringify(param)` (with param) — object params (e.g. `{ action: 'refresh' }` vs `{ action: 'check' }`) therefore get distinct keys, and param-less `modelFetch(name)` calls retain the bare `name` key.
### `modelSet(name, data)`
Set a model's data directly from a WebSocket payload — bypasses the fetch cycle (no
`fetch`, no `refreshing` flag). Directly assigns to the reactive proxy so it triggers a
re-render. Clears `model.loading` unconditionally on arrival of real data and resets
`model.error` to `null`.
```javascript
// Called by websocket.js for every WS snapshot / delta — usually you will not call this
modelSet('firewall', payload); // payload: the subsystem state object
```
| Parameter | Description |
|---|---|
| `name` | Model name (e.g., `'firewall'`). Unknown names are a no-op. |
| `data` | The full subsystem state payload from the WS `snapshot`/`versions`/`tick` message. Replaces `model.data` wholesale — pages render against the new reference. |
`websocket.js` maps subsystem → model name (`networkd``network`), and never applies a
`null` payload (a failed collector keeps the current data). See **WS Message Types** /
**WS Data Streaming Flow** below.
### `refreshByTopic(topic)`
Refresh all models whose subsystem topic matches. Called by `websocket.js` when a WS message arrives.
Refresh all models whose subsystem topic matches via `modelFetch()`. Retained for
manual / non-WS refresh paths; `websocket.js` no longer calls it (data arrives via
`modelSet` instead).
| Model `subsystem` | Topic | Match? |
|---|---|---|
@@ -363,7 +431,7 @@ html`<div class="card">
```javascript
html`<${Badge} text=${val} variant="info" />`
html`<${ConfirmDelete} url=${url} message=${msg} success="Deleted" refresh="firewall" />`
html`<${ConfirmDelete} url=${url} message=${msg} success="Deleted" />`
```
**Interpolation:** Values are interpolated with `${...}`. Use `esc()` for user-controlled text:
@@ -489,7 +557,7 @@ Pages get data from models reactive — they never call `apiFetch` in `load()`.
1. **Mount**: `init()` creates state → `load()` fires if defined → component tracked by key.
2. **Update**: Reactive state change (from model data update, navigation, etc.) → `render()` re-executes → VDOM diff patches DOM.
3. **WS auto-refresh**: Topic message arrives → `refreshByTopic()` → `modelFetch()` for matching models → `model.data` update → reactivity triggers `render()`.
3. **WS stream**: A `snapshot`/`versions`/`tick` message arrives → `modelSet()` patches the matching model in place → `model.data` update → reactivity triggers `render()`.
4. **Unmount**: `onUnmount()` called if defined → component entry destroyed.
### `hComp(renderer, key)`
@@ -562,37 +630,31 @@ socket must be closed explicitly on a terminal transition; `app.js` listens for
### WS Message Types
| Type | Fields | Effect |
|---|---|---|
| `versions` | `updated: [topic, …]` | Refresh all models matching listed topics |
| `refresh` | `topics: [topic, …]` | Same as `versions` |
| `notify` | `topic` | Refresh all models matching the topic |
| `status` | `topic` | Refresh all models matching the topic |
The daemon streams state data directly — no HTTP round-trip for auto-refresh:
Model `subsystem: '*'` matches all topics.
| Type | Fields | When sent | Effect |
|---|---|---|---|
| `snapshot` | `data: {subsystem: state \| null, …}` | Once on connect (after JWT handshake) | `modelSet()` for every subsystem; `null` payloads (failed collectors) are skipped |
| `versions` | `subsystem`, `data` | Structural change (config mutated, bump detected) | `modelSet()` for the matching model |
| `tick` | `subsystem`, `data` | Volatile-only change (e.g., `system` metrics at 1s cadence) | `modelSet()` for the matching model |
### WS Auto-Refresh Flow
Unknown or retired shapes (legacy `versions.updated` / `tick.subsystems`, `refresh`, `notify`,
`status`) are ignored — no backward compat.
When a WS message arrives for a topic:
1. `refreshByTopic(topic)` iterates registered models.
2. Matching models call `modelFetch(name)`.
3. Model fetch updates `model.data`, triggering reactivity and page re-renders.
4. In-flight dedup prevents duplicate fetches.
System name → model name mapping is handled internally (`networkd` → `network`); unknown
subsystem names fall through to the raw name.
Pages have no awareness of WS events. The model layer handles all WS-driven refresh.
### WS Data Streaming Flow
### `onMessage(topics, handler)`
When a data-carrying WS message arrives:
1. `handleMessage()` maps the subsystem to its model name.
2. `modelSet(name, data)` replaces `model.data` in place — no fetch, no `loading`/`refreshing` churn.
3. Reactivity detects the change and re-renders the pages reading that model.
4. A `null` payload is never applied — it means the collector failed and stale good data is kept.
Direct one-off subscription for code outside `definePage`:
```javascript
const unsub = onMessage(['firewall'], (msg) => {
// handle raw message
});
// Later: unsub();
```
Handler receives the parsed WS message object.
Pages have no awareness of WS events. Initial load uses `modelFetch` over HTTP (a 3-second timer
in `app.js` kicks in if no snapshot has arrived yet); afterwards the WS stream is the sole
auto-refresh path for state-backed models.
## API
@@ -633,7 +695,7 @@ function MainContent() {
### `apiSubmit(config)`
Build a form action for `formModal`. Collects body, validates, submits via `apiFetch`, toasts, and closes the modal on success. After success, refreshes the named model(s).
Build a form action for `formModal`. Collects body, validates, submits via `apiFetch`, toasts (appending an auto-synced note when the response includes a `synced` array), and closes the modal on success. Affected state-backed models update from the daemon's WS delta — no explicit `modelFetch`.
```javascript
apiSubmit({
@@ -642,7 +704,6 @@ apiSubmit({
body: () => ({ name: $val('zone-name') }),
validate: (b) => !b.name ? 'Name required' : null,
successMsg: 'Zone created',
refresh: 'firewall', // model name(s) to refresh after success
closeModal: () => closeModal(), // optional, called after success toast
}),
```
@@ -658,10 +719,13 @@ Returns an array of action descriptors matching the `formModal` action shape. Sp
| `body` | `() => body` function, or `undefined` for no body |
| `validate` | `(body) => string | null` — validation function |
| `successMsg` | Success toast message |
| `refresh` | Model name (`string`) or array of names (`string[]`) to refresh via `modelFetch()` after success |
| `closeModal` | Optional function to call after success (e.g., `() => closeModal()`) |
| `submitText` | Submit button text (default: `'Submit'`) |
> The legacy `refresh` option is no longer supported — state-backed models are
> updated by the WS delta after the mutation. To refresh a non-state model after
> success, use the `onComplete`/`onSuccess` callbacks on the wrapping component.
### `checkAbort(ac)`
**Deprecated.** Use model layer (`modelRegister` / `modelFetch`) for data fetching with abort handling and loading state management.
@@ -725,7 +789,7 @@ poll({
onErrorKey: (d) => d.status === 'failed',
onComplete: (d) => {
toast('Certificate issued', 'success');
modelFetch('acme');
// No modelFetch — the WS delta updates the acme model (state-backed).
},
onError: (d) => {
toast('Issuance failed', 'error');
@@ -824,7 +888,7 @@ Flex button container with 8px gap. Accepts VNode children directly.
```javascript
ActionGroup(
h('button', { class: 'btn btn-primary', 'on:click': addFn }, 'Add'),
ActionButton({ url: '/api/apply', label: 'Apply', refresh: 'firewall' }),
ActionButton({ url: '/api/apply', label: 'Apply' }),
)
```
@@ -889,15 +953,16 @@ Card container with optional header.
#### `ConfirmDelete(props)`
Delete button with native `confirm()` dialog, then API `DELETE` call, toast, and model refresh. Shows a spinner animation during the API call, auto-disables the button, and optionally marks the parent row/card as pending-deletion until the model refresh removes it from the DOM.
Delete button with native `confirm()` dialog, then API `DELETE` call and a success toast (appending an auto-synced note when the response includes a `synced` array). Shows a spinner during the API call, auto-disables the button, and optionally marks the parent row/card as pending-deletion. State-backed models update from the daemon's WS delta — no `modelFetch`.
```javascript
ConfirmDelete({
url: '/api/firewall/zones/myzone',
message: 'Delete zone myzone?',
success: 'Zone deleted',
refresh: 'firewall',
label: 'Delete',
deleteKey: 'myzone',
onComplete: () => { /* optional, runs after successful delete */ },
})
```
@@ -908,14 +973,15 @@ ConfirmDelete({
| `url` | API DELETE URL |
| `message` | Confirmation prompt text |
| `success` | Success toast message (default: `'Removed'`) |
| `refresh` | Model name (`string`) or array of names (`string[]`) to refresh via `modelFetch()` |
| `refresh` | **Legacy — accepted but ignored.** State models are updated by the WS delta. |
| `label` | Button text (default: `'Remove'`) |
| `body` | Optional JSON body to send with DELETE |
| `deleteKey` | Unique identifier for the item. When provided, marks the row/card as pending-deletion (opacity + red border) after API success until model refresh removes it from the DOM. Requires `_deleting.has(key)` class binding on the parent element. |
| `deleteKey` | Unique identifier for the item. When provided, marks the row/card as pending-deletion (opacity + red border) after API success; the mark is auto-purged after 2s (the WS delta normally removes the row sooner). Requires `_deleting.has(key)` class binding on the parent element. |
| `onComplete` | Callback after a successful deletion. Wire it to `modelFetch()` for non-state models. |
#### `ActionButton(props)`
Inline button that POSTs to an API endpoint, toasts on result, and optionally refreshes models. Supports toggle labels for on/off buttons. Shows a spinner animation during API calls and auto-disables the button to prevent double-submit.
Inline button that POSTs to an API endpoint and toasts on result (appending an auto-synced note when the response includes a `synced` array). Supports toggle labels for on/off buttons. Shows a spinner during API calls and auto-disables to prevent double-submit. State-backed models update from the daemon's WS delta — no `modelFetch`.
```javascript
ActionButton({
@@ -925,7 +991,7 @@ ActionButton({
label: 'Apply',
successMsg: 'Applied',
errorType: 'error', // optional, defaults to 'error'
refresh: 'dnsmasq', // model name(s) to refresh
onSuccess: () => { /* optional, runs after the success toast */ },
cls: 'btn btn-outline', // optional
disabled: false,
})
@@ -937,7 +1003,6 @@ ActionButton({
labelOn: 'Disable',
labelOff: 'Enable',
condition: z.masquerade,
refresh: 'firewall',
})
```
@@ -953,7 +1018,8 @@ ActionButton({
| `condition` | Toggle condition for `labelOn`/`labelOff` |
| `successMsg` | Success toast message |
| `errorType` | Toast type for errors (default: `'error'`) |
| `refresh` | Model name (`string`) or array of names (`string[]`) to refresh via `modelFetch()` after success |
| `refresh` | **Legacy — accepted but ignored.** State models are updated by the WS delta. |
| `onSuccess` | Callback after the success toast. Wire it to `modelFetch()` for non-state models (e.g., `backends`). |
| `cls` | Button CSS classes (default: `'btn btn-outline'`) |
| `disabled` | Disabled state |
@@ -968,8 +1034,8 @@ ActionCell({
removeUrl: '/api/proxy/domains/' + enc(d.domain),
removeMessage: 'Remove proxy for ' + d.domain + '?',
removeSuccess: 'Domain removed',
removeRefresh: 'proxy',
removeLabel: 'Delete',
deleteKey: d.domain,
})
```
@@ -982,7 +1048,7 @@ ActionCell({
| `removeUrl` | API DELETE URL |
| `removeMessage` | Confirmation prompt text |
| `removeSuccess` | Success toast message |
| `removeRefresh` | Model name (`string`) or array of names (`string[]`) to refresh after delete |
| `removeRefresh` | **Legacy — accepted but ignored.** State models are updated by the WS delta. |
| `removeLabel` | Delete button label (default: `'Remove'`) |
| `removeBody` | Optional JSON body to send with DELETE |
| `editCls` | Override classes for edit button (default: `'btn btn-sm btn-outline'`) |
@@ -1098,7 +1164,6 @@ Table({
url: '/api/item/' + enc(i.id),
message: 'Delete ' + esc(i.name) + '?',
success: 'Item removed',
refresh: 'firewall',
})),
)),
emptyText: 'No items',
@@ -1167,7 +1232,6 @@ const addZone = QuickModal({
validate: (b) => !b.name ? 'Name required' : null,
successMsg: 'Zone created', // or (data) => string
},
refresh: 'firewall', // model name(s) to refresh after success
});
// Usage in render:
@@ -1183,9 +1247,9 @@ h('button', { 'on:click': () => addZone(state) }, 'Add Zone')
| `submit.url` | API URL or `(data) => string` |
| `submit.method` | HTTP method (default: `'POST'`) |
| `submit.body` | `(data) => object`, body to send (note: the function is called with the data argument from the outer call) |
| `submit.validate` | `(body) => string | null`, validation function |
| `submit.validate` | `(body) => string \| null`, validation function |
| `submit.successMsg` | Success toast message or `(data) => string` |
| `refresh` | Model name (`string`) or array of names (`string[]`) to refresh via `modelFetch()` after success |
| `refresh` | **Legacy — accepted but ignored.** State models are updated by the WS delta after success. |
| `handler` | Optional custom handler `(data, closeModal) => void` that bypasses apiSubmit |
| `submitLabel` | Submit button label (default: `'Submit'`) |
@@ -1201,7 +1265,6 @@ const editIface = MultiSelectModal({
selected: zone.interfaces,
fieldKey: 'interfaces',
successMsg: 'Interfaces updated',
refresh: 'firewall',
});
// Usage:
@@ -1218,7 +1281,7 @@ h('button', { 'on:click': editIface }, 'Edit')
| `selected` | Currently selected values (`string[]`) |
| `fieldKey` | JSON key for the submitted field |
| `successMsg` | Success toast message (default: `'Updated'`) |
| `refresh` | Model name (`string`) or array of names (`string[]`) to refresh via `modelFetch()` after success |
| `refresh` | **Legacy — accepted but ignored.** State models are updated by the WS delta after success. |
### Toast
@@ -1247,10 +1310,10 @@ Dev mode (`VACUUM_WALL_DEV` set) disables aggressive static asset caching.
## Conventions
- **Pages** export `definePage({ … })` as default. Each page in `webui/static/pages/`.
- **Model-first data loading**: Pages get data from `getModel(name)` in `init()`. The model layer handles fetching, loading states, error handling, and WS-driven refresh. Pages never call `apiFetch` in `load()`.
- **Model-first data loading**: Pages get data from `getModel(name)` in `init()`. State-backed models are populated by the WebSocket (snapshot + per-subsystem deltas → `modelSet`); `modelFetch` is the HTTP fallback and the path for non-state models. Pages never call `apiFetch` in `load()`.
- **Render pattern**: `renderGuard` early return → data rendering. Always return VNode array or single VNode.
- **Multi-model pages**: Use `renderGuardMulti(title, subtitle, ...models)` for combined loading/error guard. `collectLoadingModels` is still exported for edge cases needing raw flags.
- **Mutation refresh**: UI components use `refresh: 'model_name'` to trigger `modelFetch()` after API mutations. Accepts single name or array.
- **Mutation updates**: UI components (`apiSubmit`, `ConfirmDelete`, `ActionButton`, `ActionCell`, `QuickModal`, `MultiSelectModal`) no longer refresh models after a mutation — the daemon re-collects the affected subsystems and the WS delta updates the models via `modelSet`. The legacy `refresh`/`removeRefresh` props are accepted but ignored. To refresh a non-state model after a mutation, pass `onComplete`/`onSuccess` wired to `modelFetch()` (e.g., `backends`).
- **Keys** on list items use unique identifiers (`item.id`), not array indices.
- **Escaping**: Use `esc()` for any user-controlled text rendered in `h()` children. Use `enc()` for URL segments.
- **Modals**: Use `formModal` + `apiSubmit` for standard CRUD operations. Use `openModal` + custom render function for non-form content.
+1 -1
View File
@@ -6,7 +6,7 @@ Vacuum Wall is a zone-based firewall appliance with a built-in SSL reverse proxy
## Architecture Overview
Vacuum Wall is built around five integrated subsystems managed through a two-layer architecture: a non-privileged Flask web UI and a privileged background daemon (`vacuum-walld`). The web UI communicates with the daemon via a Unix socket. The daemon handles all privileged operations (sudo) for the subsystems: the traffic plane uses firewalld with its nftables backend (zone-based policies, source NAT, destination NAT); the DNS/DHCP plane serves private subnets via dnsmasq; the network plane uses systemd-networkd for static IP management; the proxy plane runs nginx with automatic ACME certificates through acme.sh; and the VPN plane uses WireGuard (wg-quick) for encrypted tunnel management. All subsystems are configured and monitored through the Flask web UI, which is itself proxied through nginx with basic HTTP authentication.
Vacuum Wall is built around five integrated subsystems managed through a two-layer architecture: a non-privileged Flask web UI and a privileged background daemon (`vacuum-walld`). The web UI communicates with the daemon via a Unix socket. The daemon handles all privileged operations (sudo) for the subsystems: the traffic plane uses firewalld with its nftables backend (zone-based policies, source NAT, destination NAT); the DNS/DHCP plane serves private subnets via dnsmasq; the network plane uses systemd-networkd for static IP management; the proxy plane runs nginx with automatic ACME certificates through acme.sh; and the VPN plane uses WireGuard (wg-quick) for encrypted tunnel management. All subsystems are configured and monitored through the Flask web UI, which is itself proxied through nginx (TLS termination only — management authentication is a Flask-layer JWT, not nginx basic auth; individual proxy domains may optionally configure their own basic auth).
## Subsystems
+1 -1
View File
@@ -11,7 +11,7 @@ ACME certificate operations via `acme.sh` run as the daemon user — not as root
This design follows the principle of least privilege: only the daemon process holds sudo access, and only for explicitly enumerated commands. The WebUI user is completely isolated from sudo.
Authentication (JWT validation, token blacklist check, permission verification) is performed at the Flask layer — not the daemon. The daemon only receives requests from the Flask process via authenticated Unix socket connections. WebSocket connections to the daemon require a JWT access token as a query parameter for validation before upgrade.
Authentication (JWT validation, token blacklist check, permission verification) is performed at the Flask layer — not the daemon. The daemon only receives requests from the Flask process via authenticated Unix socket connections. WebSocket connections to the daemon require a JWT access token, sent as the raw `Sec-WebSocket-Protocol` subprotocol name (the legacy `Bearer <token>` subprotocol and an `X-Auth-Token` header fallback are also accepted), validated before the socket upgrades.
## Communication Between WebUI and Daemon
+174
View File
@@ -0,0 +1,174 @@
# State Model Reference
Authoritative reference for the shapes returned by the daemon's
pre-computed state store (`lib/state.py`), collected per subsystem and
pushed over the WebSocket (snapshot on connect, per-subsystem deltas
after every change).
Python schemas live in `lib/schema.py` (TypedDicts); each collector's
return annotation references them.
## Shared notes
- Every collector return carries a top-level `timestamp` (ISO-8601).
- Subsystems with a declarative config expose pending state as a status
dict: `status: {"pending_changes": bool}`, **except firewall**, which
uses `pending: {config_pending() result}`.
- A subsystem whose collection failed holds `null`/`None` in the state
store — WS snapshots and deltas skip `null` payloads so a failed
collector never overwrites good client data.
## State shape summary
`state_store.get(<subsystem>)` returns:
| Subsystem | Poll | Volatile fields | Top-level keys |
|---|---|---|---|
| `firewall` | 30s | `interfaces[].ips`, `interfaces[].ipv6` | `config`, `active_zones`, `interfaces`, `available_services`, `zones`, `rich_rules`, `pending`, `timestamp` |
| `dnsmasq` | 10s | *(none)* | `config`, `status`, `leases`, `timestamp` |
| `nginx` | 60s | *(none)* | `config`, `domains`, `status`, `timestamp` |
| `acme` | 300s | *(none)* | `certs`, `email`, `account`, `timestamp` |
| `wireguard` | 10s | `status.peers[].transfer_received`/`.transfer_sent`/`.latest_handshake` and the same three under `status.classes[].peers[]` | `config`, `status`, `peers`, `timestamp` |
| `networkd` | 10s | `interfaces[].addresses` | `config`, `interfaces`, `status`, `timestamp` |
| `system` | 1s | `load`, `memory`, `swap`, `traffic` | `load`, `memory`, `swap`, `traffic`, `timestamp` |
Poll intervals are overridable via `VACUUM_WALL_POLL_INTERVALS`
(`subsystem:seconds,subsystem:seconds`).
## Firewall
Top-level `FirewallState`:
```
{
config: {}, // config/firewall/config.json
active_zones: {zone: [iface]}, // zones with assigned interfaces
interfaces: [ // ip link/addr parsing
{name, mac, state, mtu, ips, ipv6, zone}
],
available_services: [str], // firewall-cmd --get-services
zones: {zone: zoneDict}, // --list-all-zones; hyphenated keys,
// may carry "sources", "ports",
// "protocols", "forward-ports", "ics",
// "icmp-blocks", "module", "rich-rules"
rich_rules: {zone: [str]}, // raw firewalld rich-rule strings,
// re-derived from zones (NO ids —
// deletion-by-id uses config.zones[].rich_rules)
pending: { // config_pending() (lib/firewall.py)
pending: [...], needs_apply: bool,
unmanaged_zones: {zone: {interfaces: [...]}}
},
timestamp: str,
}
```
Notes:
- `interfaces[].ips` / `interfaces[].ipv6` hold `"ip/prefix"` strings
(IPv6 list is separate).
- The zone dict's rich-rules key is HYPHENATED (`"rich-rules"`);
`state.rich_rules` is the snake_case top-level re-derivation.
## Dnsmasq
```
{
config: {}, // config/dnsmasq/config.json, deep-merged
status: {
service_active: bool, config_file_exists: bool,
active_leases: int, pending_changes: bool
},
leases: [
{expires, mac, ip, hostname, interface} // expires = ISO-8601 or ""
],
timestamp: str,
}
```
## Nginx
```
{
config: {}, // config/nginx/config.json
domains: [ // flattened: one entry per domain+path
{domain, path, backend, online, force_ssl, backend_name, cert,
[is_management], [is_websocket]}
],
status: {pending_changes: bool},
timestamp: str,
}
```
## ACME
```
{
certs: [ // list_certs(); extra keys possible
{domain, expiry, renewed, status, days_remaining, ...}
],
email: str,
account: {registered, email, ca, key_length},
timestamp: str,
}
```
## WireGuard
```
{
config: {}, // private_key stripped from interface
// AND every access class
status: {
up: bool, // true when ANY managed iface is up
interface: {}, peers: [], // legacy single interface (wg0)
classes: {class: {up, interface, peers}}, // per wg-<class>
pending_changes: bool
},
peers: [ // config peers, private keys stripped
{name, public_key, endpoint, allowed_ips,
persistent_keepalive, preshared_key, ...}
],
timestamp: str,
}
```
Runtime peers (`status.peers[]`, `status.classes[].peers[]`) carry:
`public_key`, `endpoint`, `allowed_ips`, `latest_handshake`,
`transfer_received`, `transfer_sent`, `persistent_keepalive`.
## Networkd
Matches `parse_networkctl_status()` output (lib/network.py) exactly:
```
{
config: {}, // config/network/config.json
interfaces: {iface: { // flat runtime entry per interface;
addresses: ["ip/prefix"], // a single combined addresses list
gateway, dns: [str], mac, // (no ipv6_addresses/routes keys)
state, link}
},
status: {pending_changes: bool},
timestamp: str,
}
```
The parser does not filter `lo`; clients that don't want it filter
client-side.
## System
Metrics only — no config, no pending state.
```
{
load: {load1, load5, load15},
memory: {total, available, used, used_pct}, // bytes; 0-100
swap: {total, used, used_pct}, // bytes; 0-100
traffic: {iface: {rx_bytes, tx_bytes,
rx_packets, tx_packets}},
timestamp: str,
}
```
All four metric fields are volatile (1s tick cadence); structural diffs
only fire on interface-set changes.