Skip to main content

Configuring Multi-Queue on Proxmox VE for OPNsense (Zenarmor)

This guide details the technical prerequisites, architectural design, and deployment steps for configuring multi-queue network packet processing on OPNsense virtual appliances running under Proxmox VE and protected by Zenarmor.

1. Architectural Overview​

Multi-Queue networking and multi-Core architecture for Zenarmor running on OPNsense platform are summarized below.

Multi-Queue Networking (VirtIO-MQ & RSS)​

Traditional virtual network adapters rely on a single pair of Receive (RX) and Transmit (TX) ring buffers. In single-queue mode, all packet handling, software interrupt processing (softIRQs), and driver I/O are pinned to a single virtual CPU (vCPU), creating a severe processing bottleneck at higher throughputs regardless of total system CPU capacity.

Multi-queue networking addresses this limitation:

  • VirtIO Multi-Queue (VirtIO-MQ): Exposes multiple parallel RX/TX queue pairs to the guest OS (vtnet driver).
  • Receive Side Scaling (RSS): Leverages a hardware/virtual hash engine to distribute network flows across different queues based on packet headers (L3/L4 IP/port 5-tuples).
  • Parallel Processing: Allows parallel execution of network stack operations across independent CPU cores, eliminating single-core saturation.

Multi-Core Worker Architecture in Zenarmor​

Zenarmor inspects network traffic using dedicated worker processes:

  • 1:1 Process-to-Queue Mapping: Zenarmor provisions a separate worker process for each active Netmap queue.
  • Core Affinity & Concurrency: If queue count exceeds available cores, thread contention and context-switching overhead significantly degrade firewall throughput.

Netmap Packet I/O Framework​

Zenarmor interfaces with the underlying network drivers via Netmap, a framework providing low-latency, zero-copy (or near-zero-copy) raw packet access directly between user space and ring buffers:

  • Native Netmap Mode: Direct hardware/virtual driver hooks bypass the FreeBSD kernel network stack entirely. Native mode is strongly recommended for multi-queue operation. Supported native interfaces include: vtnet (VirtIO), ix (Intel 10GbE), ixl (Intel 40GbE), igb (Intel 1GbE), cxgbe (Chelsio), re (Realtek), and em (Intel legacy).
  • Generic Netmap Mode: Uses standard OS packet hooks as a fallback. It incurs higher CPU overhead and does not offer multi-queue performance parity.
  • Shared Global Buffer Pool: Netmap provisions packet buffers out of a fixed, pre-allocated shared memory pool. The aggregate number of queues across all protected interfaces must not exceed this global limit to avoid memory allocation failures.

2. Resource Sizing & Planning Rules​

Before enabling multi-queue, calculate the optimal queue count (NQ) per interface using the constraints below.

Rule 1: vCPU to Queue Alignment​

Assign at least NQ dedicated vCPUs to the OPNsense VM: vCPU Count >= NQ

  • For Proxmox host configurations, use host CPU type to pass through physical CPU instruction sets and avoid overcommitting physical host cores.

Rule 2: Memory Provisioning​

Each Zenarmor worker process consumes between 300 MB and 1.5 GB of RAM depending on connection count, policy complexity, and active threat databases: Required System RAM >= NQ X 2GB

  • Example: If NQ = 4, allocate at least 8 GB of RAM to the OPNsense VM dedicated to Zenarmor and base system services.

Rule 3: Netmap Global Buffer Limits​

Netmap + Zenarmor multi-queue architecture's shared pool enforces a hard threshold on the total number of simultaneous queues across all protected interfaces:

Simultaneously Protected InterfacesMax Queues per Interface (NQ)Max Total Aggregate Queues
1 Interface<= 1212
2 Interfaces<= 816
3 Interfaces<= 618
4 Interfaces<= 520

Note: Up to 4 interfaces can be protected concurrently in multi-queue mode. The maximum selectable queue count on any single interface is 12.

Rule 4: Throughput and Traffic Hashing​

Under balanced flow distribution, plan for approximately 2 Gbps of throughput per worker/queue:

  • 10 GbE Link: NQ = 5 (requires at least 5 vCPUs and 10 GB RAM).
  • 40 GbE Link: NQ = 12 (requires at least 12 vCPUs and 24 GB RAM).

Packet distribution relies on hash algorithms:

  • In multi-queue mode, flow steering relies on an L2 hash (Source/Destination MAC addresses) or an L3/L4 hash (IP/Port tuples). Ensure traffic traversing the firewall contains diverse endpoint identifiers to prevent all traffic from clustering onto a single queue worker.

3. Pre-Configuration Shell Verification​

The configured NQ must not exceed the system/driver default queue count. Verify the active queue limit in the OPNsense shell (via SSH or Console):

# Query vtnet1 (VirtIO interface)
sysctl dev.vtnet.1 | awk '/\.rxq[0-9]+\.cpu:/ {cpu++} /\.rxq[0-9]+\.ibytes:/ {ibytes++} END {print cpu ? cpu : ibytes}'

# Query ix0 (Physical Intel 10G passthrough)
sysctl dev.ix.0 | awk '/\.rxq[0-9]+\.cpu:/ {cpu++} /\.rxq[0-9]+\.ibytes:/ {ibytes++} END {print cpu ? cpu : ibytes}'

The output integer represents the maximum available hardware/driver queue limit recognized by the kernel.

4. Implementation Steps​

Multi-Queue networking implementation steps are explained below.

Step 1: Proxmox VE VM Configuration​

  1. Open the Proxmox VE Web UI and select the OPNsense Virtual Machine.

  2. Navigate to Hardware > Processors:

    • Set Cores to a value greater than or equal to NQ (e.g., if NQ = 4, set cores >= 4).
    • Set Type to host for optimal cryptographic and packet-processing throughput.

    Figure 1. Processors Settings on Proxmox

  3. Navigate to Hardware > Network Device:

    • Select the target network interface and click Edit.
    • Ensure the model is set to VirtIO (paravirtualized).
    • Expand Advanced Options and locate the Multiqueue field.
    • Set Multiqueue to your planned NQ value (must be <= 12, and <= total assigned vCPUs).

    Figure 2. Network Device Settings on Proxmox

  4. Click OK and completely power cycle (stop and start) the VM to apply hypervisor-level queue changes.

Step 2: OPNsense & Zenarmor Configuration​

  1. Log in to the Zenconsole or OPNsense Web GUI.

  2. Navigate to Zenarmor > Settings > Interface.

  3. Locate the target interface (e.g., vtnet1 or ix0).

  4. Toggle on Enable multi-core support.

  5. Save the configuration.

    Figure 3. Zenarmor Multi-core Settings

Last updated on by Zenarmor