How to effectively manage XDP/EBPF for better DDOS protection

[Live broadcast preview] Will large models replace programmers? "

Extended Berkeley Packet Filter ( eBPF ) can be updated quickly and continuously, making it ideal for handling frequent security configuration changes.

Translated from How to Manage XDP/eBPF Effectively for Better DDoS Protection , author Ivan Koveshnikov.

The Extended Berkeley Packet Filter ( eBPF ) map serves as a high-level interface for atomic updates of the shared memory segments used as shared memory and provides a powerful configuration interface for eBPF programs. The read-copy-update mechanism minimizes performance overhead in the hot path. Additionally, eBPF mapping allows exclusive access to shared memory segments. They can handle mixed map types (arrays, hash tables, bloom filters, queues, and ring buffers), which makes them ideal for complex configurations such as security .

As configuration complexity increases, so does the need for connections between different mapping entries. If there are too many connections between map entries, the ability to make atomic configuration updates begins to degrade. Updating only one map entry may mean that other entries must be updated at the same time, which may cause inconsistencies during updates.

Apply XDP for advanced traffic management

Consider a simple eXpress Data Path (XDP) program that classifies and filters traffic based on a priority five-tuple rule set. The program processes the next packet based on the rule's priority and a combination of the packet's source IP address, destination IP address, protocol, and source and destination ports.

Flowchart of classification leading to processing.

The following is an example of a network configuration rule:

Any traffic from subnet A is always allowed.
Restrict clients in subnet C from accessing the Web server in subnet B.
Restrict access to the Web server in subnet B.
All other access is denied.

These rules require traffic classification rules and restrictions to be stored in the configuration, which can be achieved by using eBPF mappings.

Understand eBPF program configuration as a tree structure

You can visualize the configuration as a hierarchical tree, with a "configuration root" as the base. This root (possibly virtual) organizes the various configuration entities to form the active configuration. Entities are either directly connected to the root for immediate global access, or nested within other entities for structured organization.

Accessing a specific entity starts at the root and proceeds sequentially ("dereferencing" each level) until the desired entity is reached. For example, to retrieve a boolean flag from an "options" structure in a collection, you would navigate to the collection, find the structure, and then retrieve the flags.

Gcore’s approach to addressing the complexity challenges of eBPF

This tree structure provides flexibility in configuration management, including atomic swapping of any subtree, ensuring smooth transitions without disruption. However, increased complexity brings challenges. As configurations become more complex, the entries become more interconnected. It is common for several parent entries to point to a single child entry, or for an entry to play a dual role, both as a property of an entity and as part of a collection.

Modern programming languages have developed mechanisms for managing complex configurations. Developers use reference counters, mutable and immutable references, and garbage collectors to ensure safe updates. However, managing the security of these configurations does not guarantee atomicity when switching between configuration versions.

The ever-changing landscape of online traffic means security operations teams must make frequent changes to security policies. Therefore, Gcore has made quick and frequent updates to Gcore DDoS protection and incorporated important features such as a regular expression engine . We moved beyond the standard one or two updates per day for self-hosted solutions to the near-constant updates required by service providers. This requirement, often overlooked in Linux applications, has led to the adoption of eBPF technology, which enables fast, uninterrupted updates.

When exploring eBPF solutions, we must thoroughly explore strategies to ensure that our eBPF configuration is handled in the best possible way. Specifically, the limitations of eBPF mapping caused our team to rethink our configuration storage strategy.

Due to kernel security validation, eBPF map entries cannot store direct pointers to arbitrary memory segments, which requires a search key to access the map entry, slowing down the lookup process. But this shortcoming provides a benefit: it allows us to divide complex configuration trees into smaller, more manageable segments, linked directly to the configuration root. what's the result? Consistency, even during non-atomic updates.

Our findings and strategies highlight the importance of careful planning and execution of eBPF programs to optimize efficiency. So let's now turn to the specific configuration update strategies for eBPF environments and their applicability to the unique requirements and limitations of the system.

Security configuration update policy

We found three update strategies to be particularly effective in enhancing program updates while ensuring high performance and flexibility.

Update Strategy 1: Gradual Transition

A gradual update strategy means incremental configuration updates across multiple mappings. This is a useful option when processing data in one map to provide a lookup key for another map. In this case, multiple map entries need to be updated and atomic conversion is not feasible. But precise and sequential update operations enable methodical updates to the configuration. Some operations on referenced configuration subtrees become safe if performed in the correct order.

For example, in the context of classification and processing, the classification layer provides lookup keys for matching security policies, which means that update operations should follow a specific order:

It is safe to insert a new security policy because the new policy has not yet been referenced.
更新现有安全策略也是安全的，因为单独更新它们通常不会出现问题。虽然原子更新是可取的，但它没有提供显著的优势。
It is safe to update the classification layer mapping to reference the new security policy and remove references to the outdated policy.
It is safe to purge unused security policies from the configuration once they are no longer referenced.

Even without atomic updates, secure updates can be performed by properly sequencing the update process. This method works best for stand-alone mappings that are not closely related to other mappings.

We recommend performing incremental updates rather than updating the entire map at once. For example, incremental updates to hash maps and arrays are completely safe. However, this is not the case for incremental updates to a Longest Prefix Match (LPM) map, because the lookup depends on elements that are already in the map. The same problem occurs when creating a lookup key for another table requires you to operate on elements from multiple maps.

Classification layers are often implemented using multiple LPMs and hash tables, providing an example of this complexity:

Lookup flows from classification to LPM and hashing, and from classification to processing to hashing, with mapping update problem description.

Update Strategy 2: Mapping Replacement

For mappings that cannot be incrementally updated without inconsistency (such as LPM mappings), replacing the entire mapping is the best solution. To replace the eBPF program's mapping, you need a mapping of the mapping. A userspace application can create a new map, populate it with the necessary entries, and then atomically replace the old map.

The mapped mapping results in two nodes with resource isolation and replacement capabilities.

Partitioning the configuration into separate maps, each describing the settings of a single entity, provides the added benefit of resource isolation and eliminates the need to re-create the complete configuration during minor updates. The configuration of each multiple entity can be stored in a replaceable map.

This method has some disadvantages. User space needs to unpin the previous mapping to maintain the previous fixed path, because the replacement mapping cannot be pinned to the same location as the previous mapping. This is especially important for long-term programs that frequently update their configuration and rely on map pinning for stability.

Update Strategy 3: Program Replacement

The map replacement method may fail when linking multiple maps together. Updating the mapping alone may result in an inconsistent or invalid state that reflects neither the old configuration nor the expected new configuration.

To solve this problem, atomic updates should happen at a higher level. Although eBPF lacks a mechanism to atomically replace a set of mappings, mappings are usually linked to a specific eBPF program. This problem can be solved by dividing the interconnected mappings and corresponding code into separate eBPF programs linked by tail calls.

Packet pipeline to program mapping flowchart, resulting in replaceable code and mapping packages for eBPF programs.

Achieving this requires loading a new eBPF program, creating and populating the mapping for it, pinning both, and then updating the program mapping from user space. This process is more laborious than simple mapping replacement, but it allows the mapping and associated code to be updated simultaneously, thus facilitating runtime code adjustments. However, using this approach is not always particularly efficient, especially when using multiple maps and subroutines to update a single map entry in a complex program.

Error handling

Handling errors when managing eBPF can be tricky. It is important to update the configuration to prevent inconsistencies. If an error occurs during an update, it can cause chaos, so having automated backups can help reduce the need for manual fixes.

You can divide errors into two categories: recoverable errors and unrecoverable errors. For recoverable errors, if something goes wrong during an update, you can simply stop it and no changes will be made. You can fix any bug without risk.

Unrecoverable errors are a little trickier. You need to handle them with care as they can affect specific configuration entities, which can break the entire system.

It is better to organize updates by configuration entity rather than update type. This way if an error occurs it only affects a specific configuration entity and not everything at once. For example, if classification rules and security policies are defined for different network segments, it would be more efficient to update them in separate cycles based on network segment rather than by update type. This makes it easier to handle automated backups, and if an unrecoverable error occurs, you'll know exactly what the impact is. Only part of the network is configured inconsistently, while the rest is unaffected or can be quickly switched to a new configuration.

Manage eBPF program lifecycle for updates

Tracking the lifecycle of an eBPF program is critical for programs that require persistence, frequent updates, and state retention across different code instances. For example, if your XDP program requires frequent code updates while maintaining existing client sessions, managing its lifecycle effectively is critical.

For developers who want to maximize flexibility and avoid constraints, the goal should be to retain only important information between reloads—data that cannot be retrieved from non-volatile storage. This way, you can use eBPF mappings for dynamic configuration adjustments.

To make the hot code reload process more straightforward, you need to be able to distinguish between state and configuration maps, reuse the state map during reload, and repopulate the configuration map from non-volatile storage. Transitioning processing from the old program to the new program and notifying all eBPF mapping users about the changes can be a bit cumbersome.

There are two common ways to implement transitions:

Atomic program replacement : This method involves attaching an XDP program directly to a network interface and atomically swapping it out during an update. This may not be the most appropriate for large, complex eBPF programs that interact with a large number of user-space programs and maps.
A libxdp-like approach : the scheduler program links to the network interface and uses tail calls to do the processing in the next program in the program map where the actual processing is done. In addition to managing map usage and pinning, it coordinates multiple handlers, enabling fast transitions between them.

The Network Interface Card (NIC) connects to the scheduler, program map, and state map, resulting in actual program configuration.

The hot reload process quickly detects and corrects configuration issues and quickly reverts to a previous stable version if needed. For complex scenarios such as A/B testing, the scheduler can use classification tables to direct specific traffic to new versions of the XDP program.

in conclusion

Through eBPF/XDP programming, Gcore has pushed the boundaries of network security and performance optimization. Our journey demonstrates our commitment to combating emerging threats through advanced eBPF/XDP capabilities. As we continue to improve our packet processing core, we are committed to delivering cutting-edge solutions that help keep our customers' networks robust and agile.

This article was first published on Yunyunzhongsheng ( https://yylives.cc/ ), everyone is welcome to visit.