Cloudflare's bot controls are expected to help address issues such as crawlers gathering information to train generative AI. This is also recent announced a system using generative artificial intelligence create “AI Maze, a new mitigation approach that uses AI-generated content to slow down, confuse, and waste AI crawlers and other bots that do not adhere to “do not crawl” directives.
However, it said today's problems were due to changes in the database permissions system, not because of generative artificial intelligence technology, not because of DNS, and not because of what Cloudflare initially suspected – a cyberattack or malicious activity such as a “hyperscale DDoS attack.”
According to Prince, the machine learning model behind Bot management which generates bot scores for requests passing through its network, has a frequently updated configuration file that helps identify automated requests; however, “A change in the underlying behavior of the ClickHouse query that this file generates has resulted in a large number of duplicate 'function' lines appearing in it.”
The message has more details about what happened next, but changing the request caused the ClickHouse database to generate duplicate information. Because the configuration file quickly exceeded its memory limits, it disabled “the primary proxy system that handles traffic for our customers for any traffic that depended on the bot module.”
As a result, companies that used Cloudflare rules to block certain bots returned false positives and shut down real traffic, while Cloudflare customers who did not use the generated bot rating in their rules remained online.
