«

Running a Multi-Tenant WAF at the Edge

Posted by Reed Morrison

Web Application Firewalls (WAFs) are a critical layer in modern web security, providing a website’s first line of defense against vulnerabilities. WAFs can be used to defend against and notify on attempted exploits, allowing for mitigations faster than organizations can patch vulnerable software. For a global CDN, this functionality must be implemented in a way that is sensitive to performance, providing response times on the order of milliseconds. When we first introduced a WAF engine to the VDMS stack three years ago, we selected the ModSecurity Rules Engine, which we found to be first-rate for individual WAF use cases. Furthermore, ModSecurity’s support of the OWASP Core Rule Set (CRS), powerful rule language, and API access to the HTTP traffic stream in real time offered significant flexibility.

Enter waflz

However, as the number of customers using the WAF increased, we began to experience performance and resource bottlenecks. In particular, ModSecurity’s dense ruleset propagated across every customer instance, driving memory and CPU utilization up across our network, which increased operational costs. Additionally, testing and deploying new rules was difficult: the rule language was often unwieldy and difficult to write and parse. These issues, along with development complexity with the existing ModSecurity library, led to the development of waflz, an open source WAF engine, published under the Apache 2.0 license.

For VDMS, waflz is a significant improvement on ModSecurity: it consumes less memory, offers better performance, and is API-driven. waflz supports a subset of ModSecurity capabilities, the OWASP Core rulesets 2.x and 3.x, and several third-party rulesets.

Designing waflz

waflz is designed from the ground up with considerations for high performance and multi-tenancy. Where necessary, the design traded for performance over flexibility. Ultimately waflz supports a restricted subset of ModSecurity capabilities. For example, some ModSecurity directives like SecRemoteRules and inspectFile were deemed unsuitable for running on the edge for security and performance reasons.

The engine can be configured with rules in either ModSecurity format or json. Indeed, the entire WAF product was designed to be “API-first”. To this end, waflz was designed to have first-class support for json as both inputs and outputs. waflz uses Google Protocol Buffers internally to represent both configuration (including rules) and alert formats. Choosing Protocol Buffers allows for interoperability with json inputs and outputs, as well as adding strictly typed schemas for both.

Some of the principle engineering challenges in a CDN are dealing with the high concurrency, and multi-tenancy that comes from serving thousands of customers: Any edge server anywhere around the globe has to be able to process a request for any of our customers as fast as possible. Furthermore, edge server applications must provide real-time patching and processing for any given customer configuration.

Fig 1. customer config patching on an edge server

Having many WAF rulesets loaded into the memory of the running HTTP Application Server process on the edge across all of our customers quickly presented a scalability challenge. waflz addressed this issue by creating the WAF rulesets only once in memory and sharing read-only references between the customer configurations and rule customizations. Additionally, several potential performance optimizations were identified, which improved request processing times, including space and time savings to some critical internal data structures.

waflz also has built-in features that make validating engine behavior and testing one-off ModSecurity rules easier, eliminating the need for a complex standalone test harnesses.

Conclusion

waflz is part of the VDMS CDN technology stack and can operate at massive scale while enabling efficient granular rule testing and customization. Despite the challenges for CDN application, ModSecurity and the new libmodsecurity are fantastic flexible libraries, ideal for many use-cases. Indeed, in the process of developing waflz, we’ve contributed back to ModSecurity development. After a successful trial period, the new WAF engine has been running in production globally for over a year, concurrently supporting thousands of different client configurations.

If you are interested in exploring waflz functionality take a look at some of the examples over on github. If you need fast and accurate security for your website, see our web application firewall services that are delivered as part of the VDMS Cloud Security Solution and the VDMS Website Acceleration Solution.

Share