Towards a device-agnostic netmap in the FreeBSD kernel

As some might have already noticed, OPNsense’s 23.1.6 release has delivered some important kernel updates to the netmap subsystem:

From OPNsense 23.1.6 Release Notes:

The other honorable mention is the netmap work we have been doing with Zenarmor and Klara on the FreeBSD kernel side which brings bridge device support as well as a considerable improvement to the emulated mode where several packet stalls and mbuf leaks have been identified and subsequently fixed. This should have an operational impact on Suricata (IPS mode) and Zenarmor. The state is much better now but please do not hesitate to contact us about issues that you might still be having with netmap-based packet flows as the topic is a rather complex one.

The goal of this blog post is to provide some insights into what we’ve been up to and what we’re planning to do in the future in the context of improving device driver support for the netmap subsystem in the FreeBSD kernel.

Background

As the Zenarmor team, in the past, we sponsored several development efforts for the netmap project. The goal of those projects was to improve the stability of the netmap support for some specific drivers. Although the work was limited to a few widely used drivers, those developments have proved to be helpful for both Zenarmor users and Suricata and Snort IPS user community who had been using that software on the FreeBSD operating system.

Problem

Although the developments were a huge step forward for some specific ethernet adapters, the netmap and ethernet driver compatibility still continued to be an issue for the majority of the ethernet drivers.

Since Zenarmor, Suricata (IPS mode), and Snort (IPS mode) rely heavily on the netmap subsystem to function properly, any issues related to this netmap driver compatibility continued to cause issues in certain configurations, impacting the user experience for the aforementioned user community at large.

The underlying reason is that netmap mangles heavily with device drivers to be able to make maximum use of their capabilities and speed. This totally makes sense from a performance perspective; and for the original use-cases netmap has been developed for in the first place. However, this approach is very hard to maintain for use cases where portability is more important than attaining the best possible wire-speed packet performance. The reason is any single driver-specific update requires a review/update on the netmap side. Additionally, any newly introduced ethernet adapter or chipset could not be used with the IPS tools unless proper netmap support is specifically developed for the driver.

Having sponsored two netmap projects prior, it became apparent to us that trying to maintain netmap compatibility on an individual driver basis was not a sustainable solution for use-cases where portability is more important than attaining best possible packet per second performance.

Possible Solutions

We needed a mechanism that would work driver-agnostic and would perform reasonably equally well for all the ethernet drivers.

We have entertained several ideas to see if we can extend the functionality of an existing kernel mechanism to achieve what we want:

1. Improve the performance of divert sockets
2. Improve and extend packet filter hooks
3. Improve the stability and performance of the netmap-emulated driver
4. Implement a brand new kernel mechanism for this goal (BSD version of eBPF)

Teaming up for the optimal solution: netmap emulated driver

We’ve discussed these ideas with the OPNsense (Deciso) team and have decided to team up with Klara Systems to assess the situation and decide on and implement the best solution together.

After evaluating the options with Allan Jude and Tom Jones of Klara Systems, netmap emulated driver stood out as the best foundation to move forward. Netmap’s emulated driver was already driver-agnostic and we would not need any additional developments on the application side of things (Zenarmor, Suricata, Snort) since these security tools already had the netmap support built in.

On the other hand, people were reporting packet stalls or performance issues related to the emulated driver. These need to be addressed.

So it was decided that we would go with the netmap-emulated driver and the project.

The Project

After some discussion, we’ve decided to approach the project in two phases and improve the netmap emulated driver in two phases and have the below deliverables:

1. Phase 1:
   • Improving the reliability and stability of the emulated mode for all ethernet devices
   • Add emulated mode support for bridge(4), VLAN(4), lagg(4) and tun(4) pseudo interfaces
2. Phase 2:
   • Improving the performance so that its performance compares reasonably well when compared to native netmap drivers.

The project has been financially sponsored by Zenarmor and OPNsense; and the technical analysis and implementation have been done by Mark Johnson, Tom Jones, and Allan Jude of Klara Systems.

In the first phase of the project, a performance and functional analysis of netmap interfaces in emulated mode (also referred to as generic mode interfaces) revealed some functional issues which could lead to failure to complete transmission of packets, leaked memory, or complete stalls of network interfaces when using certain devices.

Buffer recycling stall

Emulated mode interfaces attempt to perform zero-copy transmission when sending packets. An in-kernel ring buffer is populated with mbuf headers to which netmap buffers are attached. When transmitting the mbuf reference count is set to 2, once this has been decremented to 1 netmap infers that the driver has freed the mbuf as part of a transmission and the transmission can be considered to be complete allowing the memory backing the mbufs to be released back to the pool. This scheme fails when software pseudo interfaces are used with netmap (such as if_lagg or if_bridge). With software interfaces the underlying hardware interface is relied on to free the mbuf during transmission completion, however, drivers are allowed to hold a small number of mbufs indefinitely. If such a buffer ends up at the tail of a netmap transmit ring further transmissions can be blocked indefinitely leading to a stall to all traffic for that netmap software interface.

To resolve this problem, a change was upstreamed to FreeBSD which removes the attempted zero-copy behavior (commit ce12afaa6fff).

As part of this change, an issue where the pkthdr field was not correctly cleared when a mbuf is released for reuse was also addressed. Before this change, some software interfaces such as if_vlan could set fields in the mbuf header which persisted when the mbuf was reused.

Leaking Packets

The networking layer is able to send packets to interfaces in batches of multiple packets by linking them together through the m_nextpkt pointer. netmap interfaces in generic mode do not handle this and when linked batches of packets were sent only the first packet from the batch would be transmitted. The remaining packets from the batch would be leaked triggering a memory leak when using interfaces in generic mode. This issue was fixed by walking the mbuf chain during transmission (commit 5f6d37787f1e).

Phase I Results:

Problems with these scenarios have been fixed. Emulated support for the mentioned pseudo ethernet adapters has been implemented.

OPNsense shipped all of these improvements with its 23.1.6 Release.

In the meantime, all of the work has been upstreamed to FreeBSD in an effort to make all of these improvements available to the wider FreeBSD community:

Future Work for Phase II: Improve emulated netmap performance

Further analysis has identified a couple of areas for future work to improve the performance of netmap and generic mode interfaces.

Following areas have been identified for future exploration. Work in these areas may be able to lead to higher performance for netmap in FreeBSD, however, higher performance might only show for certain types of network workloads or patterns of traffic.

Memory Allocation

One area to explore to improve netmap performance potentially is memory allocation. netmap offers two interfaces for memory allocation for applications:

1. The application can mmap /dev/netmap
2. The application can provide pages for netmap to use which it has allocated already using a sparsely documented feature called extmem

In the first case, netmap handles memory allocation for the buffers used to pass packets between the kernel and userspace. In this situation, it appears that netmap will always use 4KB mappings while aiming to always make mappings physically contiguous. This mechanism should enable the kernel to promote mappings to superpages when it can.

In the second case, the application has control over how the pages are allocated. This can potentially enable an application to use superpages for mappings, but the application has to be modified to support this.

Netmap’s allocator could be modified to be aware of superpages, enabling it to better size allocations to the available page sizes on a system. In certain workloads, it is possible for superpage aware handling of network data to offer a significant performance benefit. However, this benefit is highly dependent on the network traffic workload.

Buffer Management

Another area for exploration related to Netmap’s memory management is its use of buffers for storing packets. Currently, netmap takes control of an interface by hooking the ifnet’s if_input routine with generic_rx_handler. Generic_rx_handler puts received packets into a linked list which is then processed by the application when it makes a rxsync call. It is possible that this linked list isn’t cache-friendly and there could be performance advantages to moving to other data structures such as a ring buffer.

Next Steps

Memory allocation and buffer management changes are likely to offer a benefit to netmap applications.

Taking the feedback of the community into consideration; we’ll continue to monitor the performance of Zenarmor and Suricata applications with the emulated driver and assess whether the current state of the netmap emulated mode meets the reliability and performance requirements of the deployment scenarios.

Zenarmor will enable netmap Emulated Mode as the default deployment mode in one of the upcoming Zenarmor releases (Most likely 1.15; but a small probability exists that it can be shipped with 1.13.2).

Thanks

As Zenarmor and OPNsense teams, we would like to extend our special thanks and gratitudes to the OPNsense community and the wider FreeBSD community at large for testing the patches and providing feedback.

Additionally, special thanks goes to Mark Johnson, Tom Jones, Allan Jude of Klara Systems for their technical guidance and implementation; Vincenzo Maffione from the Netmap team for providing precious feedback, code review and guidance; and for and to Sabina Anja for her ongoing support during the course of the project.

Watch Now