vacuum-wall/docs/deployment.md

# Deployment Guide

This guide walks through deploying Vacuum Wall on a real appliance or server. Vacuum Wall is an SSL proxy firewall appliance that combines edge proxying, firewall management, DHCP, DNS, and WireGuard in a single device.

---

## Prerequisites

- **OS**: Clean Debian 13 (Trixie) system. Also works on Debian 12 with backports for firewalld.
- **git**: Required for cloning the repository.
- **Access**: Root access to the machine.
- **Networking**:
  - One public-facing network interface (external/edge). This receives inbound traffic and serves the management UI.
  - At least one LAN network interface (internal). This connects to your downstream network and will serve DHCP/DNS.
- **DNS**: A DNS record pointing to the appliance's public IP for the management domain (e.g., `wall.example.com`).
- **Minimum hardware**: 1 CPU, 512 MB RAM, 4 GB disk.

---

## Installation

Download the Vacuum Wall repository onto the target machine, then run the installer with required settings. All options accept both CLI flags and environment variables (CLI takes precedence).

```bash
# Production: all env vars
MGMT_DOMAIN=wall.example.com \
MGMT_PASS="strongpassword" \
./scripts/install.sh --user vacuum-wall

# Dev mode: CLI flags, auto-detects repo owner
./scripts/install.sh --dev --mgmt-pass strongpassword

# mDNS (LAN-only, no DNS record needed)
./scripts/install.sh --mgmt-domain vacuum-wall.local --mgmt-pass strongpass
```

### Options

All settings that can be passed as an environment variable also have a CLI flag equivalent. CLI flags take precedence over environment variables.

| Flag | Env Var | Required | Description |
|---|---|---|---|
| -- | `MGMT_DOMAIN` | No | Domain for the management WebUI. Auto-detected from the system hostname; defaults to `$(hostname -f \|\| hostname).local` (FQDN first, falling back to the short hostname; mDNS-served on the LAN). **Errors if the hostname is undetectable and this is not set.** |
| `--mgmt-domain` | `MGMT_DOMAIN` | No | (same as above) |
| `--mgmt-pass` | `MGMT_PASS` | Yes | Password for the initial admin user (default: `admin`). Creates the admin user in the SQLite database with full `rw` permissions on all subsystems. |
| `--mgmt-user` | `MGMT_USER` | No | Username for WebUI access. Defaults to `admin`. |
| `--user, -u` | `USER_NAME` | Yes* | WebUI service user (created if it does not exist). Required for non-dev mode. In `--dev` mode, auto-detected from repo owner. |
| `--path, -p` | `INSTALL_DIR` | No | Install directory. Defaults to repo root. Set to deploy from a custom path (e.g., `/opt/vacuum-wall`). |
| `--dev` | -- | No | Development mode: auto-detects repo owner as service user, skips safety warning. |
| `--wan-iface` | `WAN_IFACE` | No | WAN interface name. Auto-detected from default gateway. |
| `--lan-ifaces` | `LAN_IFACES` | No | LAN interface names, comma-separated. Auto-detected from non-loopback, non-WAN interfaces. |
| `--force-venv` | — | No | Force recreation of the Python virtual environment. |

Run `./scripts/install.sh --help` for full usage.

---

## Dev Mode

The `--dev` flag is designed for developers working in a git clone. It auto-detects the repo owner and uses that user as the WebUI service user.

### Ownership Model

In dev mode, the ownership model preserves the developer's ability to work with the repository:

- **Project directory**: Owned by the repo owner (e.g., `wall`), group is the repo owner's primary group (e.g., `wall`). The developer retains full control — `git add`, `git commit`, editing code and config files all work normally.
- **Daemon access**: The daemon user (`walld`, i.e. `${USER_NAME}d`) has the repo owner's primary group as its own primary group, granting read access to all project files. The project directory has the setgid bit (`g+s`) on all subdirectories, ensuring new files inherit the group.
- **`.venv/` and `data/`**: Owned by the repo owner, group is the repo owner's primary group. The developer can run `pip install`, inspect logs, and manage runtime artifacts. The daemon reads `.venv/` (Python interpreter) and writes to `data/` (runtime files) via group permissions.
- **Daemon socket** (`data/daemon.sock`): Owned by `walld:<group>` (mode `0660`). The repo owner accesses it via primary group membership.

### Running the Installer in Dev Mode

```bash
./scripts/install.sh --dev --mgmt-pass strongpassword
```

The script detects the repo owner (e.g., `wall`), creates the `walld` daemon user (`${USER_NAME}d`) with the repo owner's primary group, and sets up the ownership model described above.

### Idempotent Re-Runs

Running `--dev` again is safe. The ownership is idempotent (`chown -R` to the same owner), supplementary group membership is deduplicated by the OS, and setgid bits are applied recursively each time.

---

## Container / Custom Deployment

You can deploy Vacuum Wall in a container or at any custom path. Use `--path` (or `INSTALL_DIR`) for the mount or bind path, and `--user` (or `USER_NAME`) for whatever system user exists:

```bash
# Docker volume mount example
./scripts/install.sh --path /app/vacuum-wall --user ww-app \
    --mgmt-domain proxy.internal --mgmt-pass strongpassword
```

The systemd `.service` unit files and the sudoers whitelist are rendered from Jinja2 templates at install time, substituting `USER_NAME`, `USER_DAEMON_NAME`, `USER_GROUP`, `PROJECT_DIR`, and `ACME_HOME`. This means no hardcoded paths remain after installation.

---

## What scripts/install.sh Does

The installer performs the following steps automatically:

- **Package installation**: Installs firewalld, nginx, dnsmasq, avahi-daemon, wireguard-tools, python3, python3-pip, jq, curl, iptables, nftables, and apache2-utils.
- **WebUI user creation**: Creates the WebUI user (from `--user`) as a system user if it does not exist.
- **Shared group**: Uses the WebUI user's primary group as the shared group between both service users.
- **Daemon user creation**: Creates the daemon user `${USER_NAME}d` (the literal `vacuum-walld` only when the WebUI user is `vacuum-wall`) — a system user with `NOPASSWD` sudo access for privileged operations. Owns the daemon socket and, outside `--dev` mode, the project directory (in dev mode the repo owner keeps project ownership).
- **Python venv**: Creates the Python virtual environment and installs project dependencies. Skips if already present (use `--force-venv` to recreate).
- **acme.sh installation**: Fetches the vendored acme.sh (via `scripts/update-vendor.sh`) and installs it to `data/acme/acme.sh`, skipping if it is already present. Also installs the `system/acme-deploy.sh` deploy hook into `data/acme/deploy/acme-deploy.sh` (acme.sh only resolves hooks from its own deploy directory) and repairs ownership of the acme.sh runtime conf files under `data/acme/` — including `account.conf`, which is chmodded to `0640` — to the daemon user, so the first acme.sh run cannot fail on owner-only files.
- **Directory setup**: Creates config directories under `config/` for each subsystem's declarative JSON, and data directories under `data/` for generated files (nginx sites, dnsmasq fragments, firewall backup, WireGuard config).
- **System-directory ownership repair**: Checks top-level system directories (`/`, `/bin`, `/boot`, `/etc`, `/home`, `/opt`, `/root`, `/srv`, `/usr`, `/var`, …) for non-root ownership — some appliance images ship with system paths owned by a regular user, which trips systemd-tmpfiles' "unsafe path transition" check. Mis-owned top-level directories are chown'd to `root:root`; if deeper mis-ownership is detected, the installer warns with a full-repair command to run before re-running.
- **Static-asset permissions**: `chmod a+rX` on `webui/static/` (plus `a+x` up the parent directory chain) so nginx's `www-data` workers can serve the management UI's static assets directly from disk, regardless of checkout umask.
- **Template rendering**: Renders the systemd `.service` files and the sudoers whitelist via Jinja2, substituting `USER_NAME`, `USER_DAEMON_NAME`, `USER_GROUP`, `PROJECT_DIR`, and `ACME_HOME`. Installed systemd and sudoers files contain no hardcoded values.
- **Sudoers whitelist**: Installs a restrictive sudoers file at `/etc/sudoers.d/vacuum-walld` granting the daemon user `NOPASSWD` sudo for only the specific privileged commands needed: firewalld management (`firewall-cmd`), nginx (config test/reload, copying/removing the generated conf files), dnsmasq (restart, lease-file reads, fragment install), WireGuard (`wg`, `wg-quick`, installing `wg0.conf`), network interface queries (`ip -o link/addr show`), systemd-networkd (`networkctl` status/reload/reconfigure, managing `/etc/systemd/network`), sysctl writes, group-permission repair on the ACME home, and journal/log reads (`journalctl`, `cat /var/log/nginx/*`). Validates syntax with `visudo -cf`.
- **IP forwarding**: Enables `net.ipv4.ip_forward=1` in sysctl.conf and applies it at runtime, required for routing traffic between zones. Appends only if not already present.
- **Firewalld initialization**: Starts and enables firewalld. Opens HTTP, HTTPS, and SSH services on the public zone for management access.
- **Dnsmasq initialization**: Starts and enables dnsmasq for future DHCP/DNS serving on internal interfaces.
- **mDNS broadcast**: Enables and starts avahi-daemon so the appliance advertises its hostname (`<hostname>.local`) on the local network.
- **Self-signed certificate**: Generates a temporary self-signed X.509 certificate for the management domain via `POST /acme/self-signed` (CN set to the domain), written to `data/certs/<domain>.crt` and `data/certs/<domain>.key` — not under `data/acme/`, where acme.sh stores issued certs. Idempotent: skips generation when both files already exist.
- **Management proxy configuration**: Registers the management domain via `POST /nginx/domains/update` (falling back to `POST /nginx/domains/add`) as the special built-in `webui` backend entry (cert `selfsigned`, forced SSL). The `/` → 127.0.0.1:9090 (Flask) and `/ws` → 127.0.0.1:9091 (daemon WebSocket) mapping is derived by the daemon from the built-in webui backend — it is not passed as paths config. Then applies nginx via `POST /nginx/apply`.
- **Admin user**: Creates the admin user (default username `admin`) with the password provided via `--mgmt-pass` in the SQLite database (`data/auth.db`), with `rw` permissions on all subsystems, and writes `config/auth/config.json` (JWT + WebAuthn settings) if missing. On re-run, updates the admin password if already present. The bootstrap runs with `VACUUM_WALL_SEED_BUILTIN_ADMIN=0`, suppressing the last-resort builtin admin seed so exactly one account exists on a fresh install.
- **Initial configs**: Once the daemon socket is up, the installer writes initial state over the daemon API: `POST /acme/self-signed` (management cert), the management domain plus `POST /nginx/apply`, and firewall zone assignment — `POST /firewall/zones/interfaces` (WAN interface → `public`) and `POST /firewall/zones/services` (http/https/ssh on `public`) when a WAN interface was detected, and `POST /firewall/zones/interfaces` (LAN interfaces → `internal`) when LAN interfaces were detected. These writes are not skipped; the only skip-if-exists rule applies to the auth config (see **Admin user**).
- **System config import**: On startup, the daemon reconciles any live system configurations (dnsmasq, wireguard, networkd, nginx, firewall) with the declarative JSON configs. This prevents drift when system files were edited manually.
- **Systemd units**: Installs four units — three `.service` files are rendered from Jinja2 templates; the `.timer` is installed verbatim:
  - `vacuum-walld.service` — the privileged background daemon (aiohttp, daemon socket).
  - `vacuum-wall.service` — the Flask WebUI backend.
  - `vacuum-wall-acme.service` — the certificate renewal oneshot.
  - `vacuum-wall-acme.timer` — periodic timer that triggers cert renewals (no template variables).

  A fifth file, `system/tmpfiles.d/vacuum-wall.conf`, is installed to `/etc/tmpfiles.d/vacuum-wall.conf` and `systemd-tmpfiles --create` is run immediately — load-bearing for the hardened unit: it provisions the volatile `/run` entries the daemon needs before `vacuum-walld` spawns (restored at every boot by `systemd-tmpfiles-setup.service`).
- **Firewalld zones**: Assigns initial zones via the daemon API (the installer does not create zones directly):
  - `public` — the WAN interface is assigned here and the `http`, `https`, and `ssh` services are opened for management access (only when a WAN interface was detected).
  - `internal` — the LAN interfaces are assigned here (only when LAN interfaces were detected); no services are added at install time.
  - `vpn` — **not** created by the installer. It is managed dynamically by `lib/sync.py` only while WireGuard peers exist (interface assignment, masquerade, and rich rules), and is cleaned up again when WireGuard is deactivated.
- **Legacy nginx config cleanup**: Removes the old nginx bootstrap configs (`/etc/nginx/conf.d/vacuum-wall-map.conf` and `/etc/nginx/conf.d/vacuum-wall-mgmt.conf`), which are replaced by the daemon-generated nginx configuration.
- **Service startup**: Enables and starts/restarts nginx, the daemon (`vacuum-walld`), the WebUI (`vacuum-wall`), and the ACME renewal timer. nginx is reloaded (or restarted) to pick up any config changes.
- **ACME account**: No account registration during install. Register the account via the WebUI after first login.

### Idempotent Re-Runs

`scripts/install.sh` is fully idempotent and safe to run multiple times. Re-running the script:

- Skips the Python venv (use `--force-venv` to rebuild)
- Restarts `vacuum-walld`, `vacuum-wall`, and reloads `nginx` to pick up changes
- Preserves existing SSL certificates (skips self-signed generation if a cert exists)
- Preserves the existing auth config (`config/auth/config.json` is only written if missing — the only skip-if-exists config rule)
- Updates admin user password if changed

This makes it safe for development workflows: simply run `bash scripts/install.sh` again to update an existing installation.

---

## Post-Installation

### Verify Services

After the installer completes, confirm all services are running:

```bash
systemctl status vacuum-walld vacuum-wall nginx firewalld dnsmasq
```

Each should be active (running). The `vacuum-wall-acme.timer` should also be active (waiting).

### Access the WebUI

Open a browser and navigate to:

```
https://wall.example.com
```

Log in with the username and password you provided during installation.

### Environment Variables

| Variable | Default | Description |
|---|---|---|
| `VACUUM_WALL_DB_BACKEND` | `sqlite` | Database backend selection |
| `VACUUM_WALL_DB_PATH` | `data/auth.db` | SQLite database file path |
| `VACUUM_WALLD_SOCKET` | `data/daemon.sock` | Daemon Unix socket path (`daemon/server.py:669`) |
| `VACUUM_WALLD_WS_PORT` | `9091` | Daemon WebSocket port on `127.0.0.1` for real-time state streaming (`daemon/server.py:31`) |
| `VACUUM_WALL_POLL_INTERVALS` | built-in per-subsystem defaults | Comma-separated `subsystem:seconds` overrides for the state-poll intervals, e.g. `firewall:60,wireguard:5`; non-integer or ≤ 0 values are skipped with a warning (`daemon/server.py:34`) |
| `VACUUM_WALL_DEV` | unset (off) | Dev-mode flag: disables aggressive static-asset caching in the WebUI (`webui/server.py:86`) |
| `VACUUM_WALL_LOG_LEVEL` | `INFO` | Log level for the WebUI and daemon processes (`lib/logging.py:49`) |
| `VACUUM_WALL_EXTERNAL_IP_URL` | built-in detection | Custom URL for external-IP detection used by ACME (`daemon/handlers/acme.py:285`) |
| `VACUUM_WALL_SEED_BUILTIN_ADMIN` | `1` | Set to `0` to skip the last-resort builtin admin seed in `get_db()`; `scripts/bootstrap_auth.py` always sets this since bootstrap creates the operator user itself (`lib/db.py:307`) |

### Post-Deploy Verification

1. Confirm `config/auth/config.json` exists with JWT secret and WebAuthn RP configuration
2. Confirm `data/auth.db` exists with admin user present
3. Confirm the management server block has no server-level `auth_basic` directive — the management UI is authenticated by the Flask-layer JWT middleware, not nginx
4. Confirm `location /ws` has `auth_basic off` — the WebSocket is authenticated by the daemon via the raw-JWT `Sec-WebSocket-Protocol` subprotocol, never by nginx
5. Access the WebUI at `https://<management-domain>` — should show a login page

### Certificate Note

The initial certificate is **self-signed** and generated during installation. Your browser will show a security warning. This is expected. Once DNS is pointing to the appliance and port 80 is accessible from the internet, use the **Certs** tab in the WebUI to issue a real ACME certificate for the management domain. After issuance, go to the **Proxy** tab and click **Apply** to reload nginx with the new cert.

---

## Configuring Your First Network

After installation, the appliance has no interfaces assigned to zones and no DHCP ranges configured. Use the WebUI to set up your LAN.

### 1. Assign a LAN Interface to the Internal Zone

1. Navigate to the **Interfaces** tab.
2. From the interface list, select your LAN interface (e.g., `eth1`).
3. Assign it to the `internal` zone.
4. Click **Apply** to update the firewall configuration.

### 2. Enable NAT/Masquerade

1. Go to the **NAT** tab.
2. Enable masquerade on the `internal` zone. This allows devices on your LAN to reach the internet through the appliance's external interface.
3. Click **Apply**.

### 3. Configure DHCP

1. Go to the **DHCP** tab.
2. Click **Add Range**.
3. Specify:
   - Address range: e.g., `192.168.2.100-192.168.2.200`
   - Lease time: e.g., `12h`
   - Interface: `eth1` (or whichever interface you assigned to internal)
4. Click **Apply**. This writes the dnsmasq configuration and reloads the service.

DNS resolution will also be provided on this interface by dnsmasq, which forwards queries upstream.

---

## Adding a Proxy Domain

Vacuum Wall's primary function is proxying incoming HTTPS traffic to internal backend services.

### 1. Add the Domain

1. Navigate to the **Proxy** tab.
2. Click **Add Domain**.
3. Fill in:
   - **Domain**: The public domain name (e.g., `app.example.com`).
   - **Backend Host**: The internal IP address of the service (e.g., `192.168.2.50`).
   - **Backend Port**: The port the service listens on (e.g., `8080`).

### 2. Issue a Certificate

1. Go to the **Certs** tab.
2. Click **Issue Certificate** and enter the domain name.
3. ACME validation requires that port 80 on the appliance is reachable from the internet and that the domain's DNS A record points to the appliance's public IP.

### 3. Reload Nginx

1. Return to the **Proxy** tab.
2. Click **Apply** to write the nginx configuration and reload the service.

The proxied domain is now accessible via HTTPS at the configured domain name.

---

## Setting up WireGuard

Vacuum Wall includes integrated WireGuard server support for VPN access.

### 1. Initialize the Server

1. Navigate to the **WireGuard** tab.
2. Click **Initialize**. This generates the server's private and public keys and creates the `wg0` interface configuration.

### 2. Add a Peer

1. Click **Add Peer**.
2. Enter a peer name (e.g., `alice`).
3. Optionally set a specific AllowedIPs range for this peer (defaults to `0.0.0.0/0`).
4. Optionally set an **Endpoint** if you know the peer's static public IP (restricts incoming connections to that IP).
5. Click **Add**. The peer's public key and preshared key are generated automatically.

### 3. Activate the Tunnel

1. Click **Apply** to write the WireGuard configuration and bring up the `wg0` interface.

### 4. Download Client Configuration

1. In the peer list, use the peer actions menu to download the client configuration file for the peer.
2. Install this configuration on the client device.

### 5. Assign WireGuard to a Firewall Zone

1. Navigate to the **Interfaces** tab.
2. Assign `wg0` to the `vpn` zone.
3. The `vpn` zone allows all traffic by default (target ACCEPT). Adjust firewall rules as needed to restrict VPN access to specific services.

### 6. Configure Firewall Rules for VPN Traffic

1. Go to the **Firewall** tab or use the **NAT** tab.
2. Add rules as needed to control what VPN peers can access. For example, you can restrict VPN peers to only reach specific internal services rather than the entire LAN.
3. Optionally enable masquerade on the `vpn` zone to allow VPN clients to reach the internet through the appliance.

---

## Troubleshooting

### Services Won't Start

Check service logs and configuration:

```bash
journalctl -u vacuum-walld --no-pager -n 50
journalctl -u vacuum-wall --no-pager -n 50
nginx -t
```

Both units also keep journal output on disk under `/var/log/vacuum-wall/` (`LogsDirectory=vacuum-wall` on both units). Nginx writes per-domain access/error logs to `/var/log/nginx/wall_mgmt_*.log` for the management domain and `/var/log/nginx/<domain>_*.log` for each proxy domain.

Common causes include port conflicts (another service on port 80/443, 9090, or 9091 — the daemon's WebSocket port), missing dependencies, or file permission issues on `data/`.

### Firewall Rules Not Applying

Verify that firewalld is running:

```bash
firewall-cmd --state
systemctl status firewalld
```

If firewalld is not running, start it with `systemctl start firewalld`. Check that the sudoers whitelist is valid:

```bash
visudo -cf /etc/sudoers.d/vacuum-walld
```

### Certificate Issuance Fails

ACME validation via the ACME provider requires:

- The domain's DNS A record points to the appliance's public IP.
- Port 80 (HTTP-01 challenge) is accessible from the internet on the external interface.
- An ACME account is registered (check the Certs page in the WebUI). Verify with:

```bash
su -s /bin/bash "$USER_DAEMON_NAME" -c "~/data/acme/acme.sh --list"
```

If port 80 is blocked or the DNS record hasn't propagated yet, wait and retry. The ACME timer will also attempt renewal automatically.

### DHCP Not Working

Verify that:

- The LAN interface is assigned to a firewalld zone (check the **Interfaces** tab or `firewall-cmd --get-active-zones`).
- Dnsmasq is running: `systemctl status dnsmasq`.
- A DHCP range is configured for the correct interface. Check dnsmasq config at `data/dnsmasq/`.
- The firewall allows DHCP traffic on the internal zone: `firewall-cmd --zone=internal --list-services` should include `dhcp` and `dns`.

### Locked Out of WebUI

If you lose access to the admin account, you can reset the password directly via SQLite:

```bash
# Stop the services
sudo systemctl stop vacuum-wall vacuum-walld

# Reset password (replace 'newpassword' with desired password)
sqlite3 data/auth.db "UPDATE users SET password_hash='NEW_HASH_HERE' WHERE username='admin';"
```

The password hash must be an Argon2id hash. You can generate one:

```bash
python3 -c "from lib.password import hash_password; print(hash_password('newpassword'))"
```

Alternatively, use the SQLite prompt to directly inspect and modify user data:

```bash
sqlite3 data/auth.db ".tables"
sqlite3 data/auth.db "SELECT username FROM users;"
sqlite3 data/auth.db "SELECT * FROM permissions WHERE username='admin';"
```

### Database Corruption

If the SQLite database becomes corrupted:

1. Stop the services: `sudo systemctl stop vacuum-wall vacuum-walld`
2. Inspect: `sqlite3 data/auth.db "PRAGMA integrity_check;"`
3. If the file is unrecoverable, delete it (`rm data/auth.db`) and start the services: `sudo systemctl start vacuum-walld vacuum-wall`. The schema is recreated on startup; if the users table is empty, the last-resort builtin admin is seeded with a random password written to `/var/log/vacuum-wall/auth.log`.
4. Re-set the password via the WebUI, or use the SQLite steps under "Locked Out of WebUI".

### WebUI Not Accessible

1. Verify nginx is running: `systemctl status nginx`.
2. Test nginx configuration: `nginx -t`.
3. Check the management proxy domain configuration via the WebUI Proxy tab, or by inspecting `config/nginx/config.json`.
4. Ensure the daemon is running and its Unix socket exists: `systemctl status vacuum-walld` and `ls -l data/daemon.sock` — the WebUI proxies every API call through this socket.
5. Ensure the WebUI service is listening on port 9090 (`ss -tlnp | grep 9090`) and the daemon's WebSocket endpoint on port 9091 (`ss -tlnp | grep 9091`).
6. If using the self-signed cert, confirm your browser trusts it or use the WebUI to issue a real ACME certificate.

---

## Reference

| Component | Service | Config Location |
|---|---|---|
| Daemon (privileged) | `vacuum-walld.service` | `daemon/` |
| WebUI backend | `vacuum-wall.service` | `webui/` |
| Reverse proxy | `nginx` | `config/nginx/config.json` (via daemon) |
| Firewall | `firewalld` | Managed via WebUI and `firewall-cmd` |
| DHCP/DNS | `dnsmasq` | `config/dnsmasq/` |
| VPN | wireguard-tools | `config/wireguard/` |
| Certificates | `vacuum-wall-acme.timer` | `$PROJECT_DIR/data/acme/` |
| Sudoers (daemon) | — | `/etc/sudoers.d/vacuum-walld` |