Architectural Zero-Downtime Migration: Staging and Cutting Over OPNsense on a Proxmox Edge Node

  1. Architectural Overview & The Zero-Downtime Objective

1.1. The Staging Challenge & Perimeter Isolation

Migrating perimeter infrastructure—including stateful firewalling, inter-VLAN routing, authoritative DNS resolution, and network monitoring—presents a fundamental operational challenge: modifying the primary edge gateway typically disrupts connectivity for all downstream production workloads and local network clients.

The objective of this architectural migration is to transition the edge routing and security stack from an over-provisioned primary compute node (Din, running on a Beelink N95) to a dedicated perimeter edge node (Nayru, running Proxmox VE 8 on a Beelink ME Mini N150 with onboard 64GB eMMC storage). This migration must be executed out-of-band with zero downtime for household and production LAN traffic during the staging, configuration, and verification phases.

[!NOTE] Note on Addressing Notations To keep this architectural guide generic, modular, and reusable across varying network environments, active IP addressing is represented using functional notation placeholders:

  • [LAN Subnet CIDR] — The primary local subnet block (e.g., /24).
  • [LAN Gateway IP] — The internal interface address and default gateway assigned to OPNsense.
  • [Edge Host IP] — The host management IP assigned to Nayru.
  • [App Host IP] — The host management IP assigned to Din.
  • [Primary DNS IP] — The static internal IP assigned to AdGuard Home ([DNS CT]).
  • [Network Sentry IP] — The static internal IP assigned to NetAlertX ([SENTRY CT]).
  • [Reverse Proxy IP] — The static internal IP assigned to Nginx Proxy Manager ([PROXY CT]).
  • [Switch Management IP] — The static IP assigned to the managed network switch.
  • [Dynamic DHCP Pool] — The address range handed out by OPNsense for dynamic client leases.

1.2. Workload Partitioning & Target Node Roles

Decoupled edge architecture requires strict workload isolation. Compute-heavy application workloads remain on the primary compute node, while edge security and routing functions are consolidated onto the dedicated perimeter host.

Target Node Node Role & Hardware Base Allocated Workloads & Guests Nayru Perimeter Edge Host
(Beelink ME Mini N150, 64GB eMMC) • OPNsense Virtual Machine (Router / Firewall)
• AdGuard Home ([DNS CT] - Authoritative DNS)
• NetAlertX ([SENTRY CT] - Network Sentry)
• opnsense-jumphost ([JUMPHOST CT] - Isolated Staging Client) Din Application Compute Host
(Beelink N95, Local Storage) • Nginx Proxy Manager ([PROXY CT] - Reverse Proxy)
• Transmission ([DOWNLOADER CT] - Downloader)
• TorrServer (Media Streaming)
• Application Containers: Speedtest-Tracker ([SPEEDTEST CT]), Vaultwarden ([VAULT CT]), Authelia ([SSO CT]), Gotify ([GOTIFY CT]), Home Assistant ([HASS CT]), FreshRSS ([RSS CT]), Actual Budget ([BUDGET CT]), OpenWebUI ([AI CT]), Dawarich ([GPS CT])

To segregate perimeter ingress from host management and internal switching, Nayru relies on dual physical network interfaces mapped to distinct Linux bridges:

  • vmbr0 (Internal LAN Bridge): Bound to Physical NIC 1 (Intel i226-V). Hosts the Nayru management interface ([Edge Host IP]) and handles internal switching between LXCs, VMs, and the physical switch.
  • vmbr1 (WAN Bridge): Bound to Physical NIC 2 (Intel i226-V). Configured strictly as an unnumbered bridge with no host IP address or default gateway, reserved exclusively for virtual network interface (vNIC) passthrough to OPNsense’s WAN port.
  1. Out-of-Band Staging & The Double-NAT Strategy

2.1. Air-Gapped Virtual Bridge Topology

Pre-configuring OPNsense without impacting active network clients requires complete logical and physical isolation from the active production switch. This is achieved using a double-NAT staging topology:

  1. Air-Gapped Internal Bridge (vmbr0): Physical NIC 1 on Nayru remains physically disconnected from the main network switch during pre-configuration. All inter-guest traffic inside vmbr0 is switched strictly within hypervisor memory space.
  2. Nested Upstream WAN Link (vmbr1): Physical NIC 2 on Nayru is connected via an Ethernet patch cable to an open LAN port on the active ISP router. OPNsense’s WAN interface is configured to acquire an IP address via DHCP from the existing gateway.

In this topology, OPNsense treats the active production network as its external ISP provider. Outbound packets generated inside the vmbr0 virtual bridge undergo Network Address Translation (NAT Layer 1) at the OPNsense VM, and subsequently undergo a second translation (NAT Layer 2) at the primary ISP router before egressing to the internet. Active clients on the production LAN remain completely unaware of the staged [LAN Subnet CIDR] environment.

[ Public Internet / Fiber ONT ] │ ▼ [ Existing ISP Gateway ] ──► (Active Production LAN: Uninterrupted) │ (LAN Port) ▼ [ Nayru: Physical NIC 2 ] │ ▼ ┌────────────────── Nayru (Perimeter Host) ─────────────────────┐ │ │ │ [ OPNsense VM: WAN Interface (DHCP from ISP Router) ] │ │ │ │ │ ▼ (NAT Layer 1 / Stateful Inspection) │ │ [ OPNsense VM: LAN Interface ([LAN Gateway IP]) ] │ │ │ │ │ ├────── [ AdGuard Home: [DNS CT] ] │ │ ├────── [ NetAlertX: [SENTRY CT] ] │ │ └────── [ Jumphost: [JUMPHOST CT] (DHCP Client) ] │ │ │ │ vmbr0 (Air-Gapped Virtual Bridge / Physical NIC 1 Unplugged)│ └───────────────────────────────────────────────────────────────┘

2.2. Isolated Jump Host Container ([JUMPHOST CT]: opnsense-jumphost)

To inspect, manage, and validate services inside vmbr0 without attaching a physical workstation to Nayru, an isolated test container ([JUMPHOST CT]: opnsense-jumphost) is deployed directly onto the internal bridge.

[JUMPHOST CT] is attached exclusively to vmbr0 and configured for dynamic IPv4 addressing. When the OPNsense VM starts inside vmbr0, [JUMPHOST CT] receives an IP lease from the staged [Dynamic DHCP Pool] along with [LAN Gateway IP] as its default gateway and [Primary DNS IP] as its primary resolver. [JUMPHOST CT] functions as an in-line synthetic client, allowing complete verification of routing tables, DNS interception policies, and firewall rules prior to physical cutover.

  1. Service Configuration & Validation via Local SSH Tunnels

3.1. Risk-Free UI Access via Local Tunneling

Because Physical NIC 1 (vmbr0) is air-gapped from the production switch, the web management interfaces bound to vmbr0 (OPNsense, AdGuard Home, and NetAlertX) are inaccessible via direct LAN routing.

To manage these interfaces securely from the operator workstation (poe-xps), Nayru acts as the SSH tunnel endpoint listening on the hypervisor host interface. SSH port-forwarding tunnels forward local workstation loopback sockets across Nayru directly to the isolated guest IPs inside vmbr0:

Establish local SSH port-forwarding tunnels through Nayru into the isolated vmbr0 bridge

ssh -L [Local Web Port]:[LAN Gateway IP]:[HTTPS Port]
-L [Local DNS Port]:[Primary DNS IP]:[DNS Web Port]
-L [Local Sentry Port]:[Network Sentry IP]:[Sentry Port]
root@[Edge Host IP]

With the SSH session active, local browser instances on poe-xps securely map to the air-gapped administrative interfaces:

3.2. Core Service Pre-Configuration & Validation

While connected through the SSH tunnel, edge services are configured and validated inside the isolated sandbox:

  • DHCP Scope Allocation: The dynamic client pool is bounded strictly within [Dynamic DHCP Pool]. Lower and upper static address blocks ([LAN Gateway IP], [Edge Host IP], [App Host IP], [Reverse Proxy IP], [Primary DNS IP], and [Network Sentry IP]) are excluded from dynamic assignment to eliminate potential IP collisions.
  • Authoritative DNS Assignment: OPNsense’s DHCP daemon is configured to hand out [Primary DNS IP] (AdGuard Home) as the absolute DNS resolver for all dynamic clients.
  • Rogue DNS Interception (Port 53 Redirection): To prevent IoT firmware or hardcoded client software from querying public DNS resolvers (e.g., 8.8.8.8), a NAT Port Forwarding rule is enabled in OPNsense:
    • Interface: LAN (vmbr0)
    • Protocol: TCP/UDP
    • Destination Port: 53 (DNS)
    • Invert Destination: Enabled (![Primary DNS IP])
    • Redirect Target IP: [Primary DNS IP]
    • Redirect Target Port: 53
  • NetAlertX LXC Kernel Capability Injection: NetAlertX ([SENTRY CT]) relies on diagnostic binaries (such as nmap, arp-scan, and fping) to perform network discovery. Inside standard Proxmox LXC instances, these binaries drop execution silently due to missing raw socket privileges. To resolve this, raw socket access must be explicitly granted in the container configuration file (/etc/pve/lxc/[SENTRY CTID].conf) on Nayru:

lxc.cap.keep: net_raw net_admin net_bind_service

  • Sandbox Egress Verification: From inside [JUMPHOST CT] (accessed via pct enter [JUMPHOST CTID] on Nayru), validation tests verify end-to-end routing, DNS interception, and double-NAT outbound transit:

Verify internal gateway reachability

ping -c 3 [LAN Gateway IP]

Verify AdGuard Home DNS resolution and rewrite responses

dig @[Primary DNS IP] google.com

Confirm rogue DNS port 53 redirection rule intercepts external queries

dig @8.8.8.8 google.com

Verify end-to-end double-NAT outbound web reachability

curl -I https://google.com

  1. Critical Operational Pitfalls & Pre-Cutover Guardrails

4.1. Layer 2 Rogue DHCP Lease Collisions

The most severe failure mode during perimeter staging is a Layer 2 DHCP collision. If Physical NIC 1 (vmbr0) on Nayru is patched into the main switch while OPNsense’s DHCP daemon is active on the same physical broadcast domain, two independent DHCP engines (the existing ISP router and OPNsense) will compete to answer DHCP DISCOVER requests.

If OPNsense wins the race condition against production clients, it will issue leases containing default gateway pointers to [LAN Gateway IP]. Because physical routing paths are not yet cut over, affected client devices will instantly lose all external WAN reachability and local network access.

Guardrail: Physical NIC 1 (vmbr0) must remain physically disconnected from the main switch throughout the staging process. The OPNsense DHCP daemon must remain bound strictly to the air-gapped vmbr0 bridge until the legacy router’s DHCP engine is powered down during cutover.

4.2. The RFC1918 WAN Filtering Trap

OPNsense enforces a default security rule on its WAN interface: “Block private networks (RFC 1918)”.

During double-NAT staging behind an existing router, OPNsense’s WAN interface receives an upstream private IPv4 address from the active gateway. If the RFC 1918 filter remains active, OPNsense will drop all ingress packets originating from the upstream router, preventing [JUMPHOST CT] from downloading package updates or establishing outbound connections.

Guardrail: During staging, navigate to Interfaces → WAN in OPNsense and uncheck “Block private networks (RFC 1918)”. Once physical cutover occurs and Physical NIC 2 (vmbr1) connects directly to the Fiber ONT (receiving a public IP address), re-evaluate and re-enable this setting to filter spoofed ingress traffic.

4.3. Split-Brain Auto-Start Prevention & MAC Address Flapping

Cross-node guest migration via vzdump backup and qmrestore/pct restore operations creates identical duplicate instances of virtual machines and containers across both hypervisors (Din and Nayru).

Proxmox VE preserves the original virtual MAC addresses of restored guest NICs. If Din reboots unexpectedly while the migrated guests on Nayru are live, Proxmox VE’s default auto-start configuration will boot the source instances on Din. This triggers a catastrophic split-brain condition: identical IP and MAC addresses appear simultaneously on separate switch ports. The managed switch will experience continuous MAC address table flapping, resulting in severe packet loss, dropped TCP connections, and network instability across the LAN.

Guardrail: Immediately upon restoring guests onto Nayru, explicitly quarantine the source instances on Din and disable target auto-start flags on Nayru until cutover:

Execute on Din to quarantine source instances and prevent boot collisions

qm set <SOURCE_VMID> —onboot 0 pct set [DNS CTID] —onboot 0 pct set [SENTRY CTID] —onboot 0 pct set [JUMPHOST CTID] —onboot 0

Execute on Nayru to keep staged instances stopped during host reboots prior to cutover

qm set <TARGET_VMID> —onboot 0 pct set [DNS CTID] —onboot 0 pct set [SENTRY CTID] —onboot 0

4.4. Physical Re-Patching & Router AP Mode Conversion

Executing physical cutover requires a strict sequence of physical layer adjustments to avoid physical bridge loops or unexpected broadcast offers:

  1. Fiber ONT Realignment: Unplug the Ethernet line coming from the Fiber ONT/Modem from the legacy ISP router’s WAN port and plug it directly into Physical NIC 2 (vmbr1) on Nayru.

  2. Perimeter Patching: Run a patch cable from Physical NIC 1 (vmbr0) on Nayru into Port 1 of the managed switch (TL-SG108E). Ensure Din is attached to Port 2.

  3. Legacy Router AP Mode Conversion: Access the administrative interface of the legacy router, disable its DHCP server entirely, and convert the hardware into Access Point (AP) / Bridge Mode. If native AP mode is unsupported by the firmware, disable DHCP, unbind its WAN port, and patch a LAN port of the old router directly into the managed switch to serve strictly as a layer-2 wireless bridge.

  4. Post-Cutover Execution, Command Center Updates & Decommissioning

5.1. Physical Cutover Execution Checklist

Execute the migration cutover systematically using this step-by-step checklist:

  1. Re-IP Host Management: Update /etc/network/interfaces on Din to set its host IP to [App Host IP] with gateway set to [LAN Gateway IP]. Apply via ifreload -a.
  2. Verify Source Quarantine: Confirm that auto-start is disabled (—onboot 0) on Din for the OPNsense source VM, [DNS CT] (AdGuard), [SENTRY CT] (NetAlertX), and [JUMPHOST CT] (Jumphost). Confirm all source instances are STOPPED. (Note: Compute applications such as [SPEEDTEST CT] Speedtest-Tracker remain active on Din and must not be quarantined).
  3. Inject LXC Kernel Capabilities: Ensure /etc/pve/lxc/[SENTRY CTID].conf on Nayru contains lxc.cap.keep: net_raw net_admin net_bind_service prior to initial startup.
  4. Execute Physical Cabling: Move the Fiber ONT patch drop to Physical NIC 2 (vmbr1) on Nayru. Connect Physical NIC 1 (vmbr0) on Nayru to Port 1 of the managed switch.
  5. Reconfigure Wireless AP & Flush Client Leases: Convert the legacy ISP router to Access Point mode (DHCP server disabled). Power-cycle the AP to flush internal client association tables and force all wireless endpoints to drop and request fresh DHCP leases.
  6. Engage Target Edge Stack: Power on the OPNsense VM, [DNS CT] (AdGuard), and [SENTRY CT] (NetAlertX) on Nayru. Enable the OPNsense DHCP daemon on the LAN interface.
  7. Set Target Auto-Start Flags: Enable persistent boot execution for the live edge stack on Nayru:

qm set <TARGET_VMID> —onboot 1 pct set [DNS CTID] —onboot 1 pct set [SENTRY CTID] —onboot 1

  1. Verify Tailscale Overlay Integrity: Check that the Tailscale sidecar daemon inside [DNS CT] reconnects cleanly to the mesh:

pct exec [DNS CTID] — tailscale status

  1. Update Reverse Proxy Ingress Targets: Log into Nginx Proxy Manager ([PROXY CT] on Din at [Reverse Proxy IP]) and update proxy forward target definitions:
  • edge.lab.example.com → https://[Edge Host IP]
  • app.lab.example.com → https://[App Host IP]
  • btop.lab.example.com → http://[Edge Host IP]
  • netalertx.lab.example.com → http://[Network Sentry IP]
  1. Update Authoritative DNS Rewrites: Confirm AdGuard Home ([DNS CT] at [Primary DNS IP]) contains active internal DNS rewrites:
  • edge.lab.example.com → [Edge Host IP]
  • app.lab.example.com → [App Host IP]
  • *.lab.example.com / *.home.example.com → [Reverse Proxy IP]
  1. Flush Client ARP/DHCP Caches: Flush local workstation ARP tables (ip neigh flush all on Linux) or renew client DHCP leases to force immediate adoption of [LAN Gateway IP] and [Primary DNS IP].

5.2. Telemetry & Dashboard Alignment (labpage)

With perimeter routing established on Nayru, the central command center (labpage, hosted at https://home.lab.example.com strictly within the Tailscale mesh overlay) must be updated to aggregate dual-node hypervisor telemetry.

  1. Dual-Node Proxmox API Metrics: Configure labpage backend integrations with API tokens generated across both nodes. Update endpoint definitions to target https://edge.lab.example.com and https://app.lab.example.com simultaneously, displaying CPU, RAM, and eMMC/NVMe usage stats side-by-side.
  2. Live btop Terminal Stream: Update the embedded terminal card configuration to ingest the live WebSocket stream from https://btop.lab.example.com (routed via NPM to [Edge Host IP] with Authelia SSO forward-authentication).
  3. Service Telemetry Verification: Confirm live health check widgets successfully poll API endpoints for AdGuard Home ([Primary DNS IP]), NetAlertX ([Network Sentry IP]), Transmission ([DOWNLOADER CT]), and active application compute containers on Din without triggering SSO redelivery loops.

5.3. Legacy Decommissioning & Storage Reclamation

Following a 24- to 48-hour burn-in period, execute non-interactive decommissioning commands on Din to permanently remove inactive source guests and reclaim storage space.

Step 1: Permanently Destroy Inactive Source Guests on Din

Run these commands from root@din to purge duplicate instances:

Destroy legacy source OPNsense VM and purge disk volumes

qm destroy <SOURCE_VMID> —purge

Destroy migrated edge containers and rootfs volumes

pct destroy [DNS CTID] —purge pct destroy [SENTRY CTID] —purge pct destroy [JUMPHOST CTID] —purge

Step 2: Prune Staging Backup Archives

Clear temporary vzdump archives from /var/lib/vz/dump/ across both hosts:

Execute on Din to reclaim local storage

rm -f /var/lib/vz/dump/vzdump-qemu-<SOURCE_VMID>-* rm -f /var/lib/vz/dump/vzdump-lxc-[DNS CTID]-* rm -f /var/lib/vz/dump/vzdump-lxc-[SENTRY CTID]-* rm -f /var/lib/vz/dump/vzdump-lxc-[JUMPHOST CTID]-*

Execute on Nayru to remove staging transfer archives

rm -f /var/lib/vz/dump/vzdump-*

Step 3: Audit Retained Workloads on Din

Verify that Din maintains strictly its assigned compute stack: Transmission ([DOWNLOADER CT]), Nginx Proxy Manager ([PROXY CT]), Speedtest-Tracker ([SPEEDTEST CT]), Vaultwarden ([VAULT CT]), Authelia ([SSO CT]), Gotify ([GOTIFY CT]), Home Assistant ([HASS CT]), FreshRSS ([RSS CT]), Actual Budget ([BUDGET CT]), OpenWebUI ([AI CT]), Dawarich ([GPS CT]), and TorrServer.

  1. Key Takeaways for Zero-Downtime Firewall Migrations

[!IMPORTANT] 1. Validate Out-of-Band via Air-Gapped Jump Hosts Never perform initial routing and firewall configuration directly on an active production LAN. By isolating the internal bridge (vmbr0) and deploying an in-line jump host container ([JUMPHOST CT]), firewall policies, DNS interception rules, and DHCP scopes can be thoroughly verified using SSH port-forwarding tunnels without risking broadcast domain collisions.

[!IMPORTANT] 2. Leverage Double-NAT for Pre-Cutover Verification Treat the existing production network as an upstream WAN provider during staging. Disabling RFC 1918 private network filtering on the staged WAN interface allows complete outbound verification, DNS testing, and package installation behind the active gateway prior to physical cutover.

[!IMPORTANT] 3. Enforce Strict Auto-Start Quarantine (—onboot 0) Standalone cross-node migrations preserve identical virtual MAC addresses. Explicitly disabling auto-start (—onboot 0) on duplicate source guests prevents severe split-brain conditions, duplicate IP assignments, and switch MAC address flapping if the source node reboots unexpectedly.

[!IMPORTANT] 4. Decouple Application Ingress via Reverse Proxying and SSO Utilizing a centralized reverse proxy (Nginx Proxy Manager) paired with wildcard DNS rewrites decouples application endpoints from underlying IP addresses. Moving or re-IPing backend containers requires updating a single proxy host target—client bookmarks, browser SSO session cookies, and external dashboard links remain entirely unaffected.