A security rule stopped websites
A web protection change caused a Cloudflare outage. A lesson in performance testing, gradual deployment and prepared rollback.
On 2 July 2019, a Cloudflare WAF rule update caused a global outage. A regular expression exhausted CPU capacity. The rule was in simulation mode: it did not block requests, but it still had to analyse them.
Protection had to return without the fault
The team stopped the WAF, restoring traffic. After checking the fix on a limited subset of traffic, it re-enabled protection. The report identified excessive regular-expression backtracking as the cause of CPU exhaustion.
Simulation still does work
NetOpiekun’s lesson: observe-only mode does not necessarily mean no service impact. A rule can still consume device or application resources. Evaluate both correctness and processing cost.
A small change can have a large reach.
Start with a small scope
We recommend recording a baseline before a change: response time, errors and resource use. Deploy to a limited scope and observe the same measures. Expand only after reviewing the result.
Testing should include unusual and longer inputs, not just a few valid examples. Agree stop conditions before deployment, rather than under incident pressure.
Rollback belongs in the deployment plan
Prepare the previous configuration, name the decision-maker and verify management access if the protected service fails. Saving a file does not prove it can be restored quickly.
Disabling protection was a response to this particular incident, not a general recommendation. Prefer the narrowest suitable rollback of the faulty change, then verify that required safeguards work after service returns.
About this account
Based on the public report named below. The account concerns the period covered by that report; NetOpiekun recommendations are identified separately. This is not our client work or a description of the organisation’s current systems.
Sources: Cloudflare — Details of the Cloudflare outage on July 2, 2019, 12 July 2019