AI crawlers that collect web content are increasing, while AI agents are beginning to go beyond reading information and interact with websites on behalf of users. As a result, the distinction between different bot roles is becoming less clear. 

For security teams, the task is no longer limited to detecting and blocking bots. They also need to decide which AI-driven automated traffic should be allowed as legitimate access and which traffic should be restricted or blocked.

As AI becomes another type of user of web services, these decisions increasingly need to be reflected in security policies.

AI Crawlers That Collect Content

An AI crawler is a bot that automatically visits web pages and collects content.

While traditional search engine crawlers index web pages for search results, AI crawlers may use web content to train generative AI models or support AI-based search and answer services.

However, not all AI crawler traffic should be treated in the same way. Rather than classifying traffic as malicious simply because it comes from an AI crawler, organizations need to set policies based on factors such as the crawler’s identity and purpose, the resources it accesses, and its behavior.

For example, a company may allow a crawler if it wants its content to appear in an AI search service. On the other hand, access may need to be restricted if the company does not want certain content used for AI training or if excessive requests affect service operations.

From Bots That Read to Agents That Act

AI-driven automation is expanding beyond content collection.

While traditional crawlers visit multiple pages to retrieve information, agentic AI can take a user’s goal, determine the tasks required, and carry out multiple steps.

When these AI agents connect to web services, they can do more than read information. They can search, use specific functions, and interact with services on behalf of users.

This creates a new challenge for security teams.

Blocking all AI agent activity simply because it is automated could also restrict legitimate use. At the same time, trusting traffic simply because it comes from an AI agent could allow unintended access or excessive activity.

As a result, traditional distinctions such as human versus bot or good bot versus bad bot are no longer enough to determine whether traffic is legitimate.

A New Approach to Managing AI Traffic

This shift also changes the purpose of bot management.

The goal is not to block AI crawlers and agents across the board, but to distinguish between automated traffic that should be allowed and traffic that needs to be managed according to the organization’s service policies.

For example, AI-driven traffic may require different policies:

•     Allow AI crawlers that are trusted and aligned with the purpose of the service
•     Rate-limit approved crawlers if they generate excessive requests
•     Restrict or block automated collection of content that should not be accessed
•     Monitor repetitive or abnormal AI agent behavior separately
•     Block known malicious bots and requests that show attack patterns

Bot management therefore needs to go beyond identifying whether traffic is automated and consider the identity and behavior of the traffic when applying policies.

Would a WAF Still Be Necessary If an AI Agent Is Legitimate? 

Verifying a bot’s identity and determining whether each request is safe for an application are separate issues. Distinguishing legitimate AI crawlers and agents and applying access policies is therefore not enough to protect an application.

For example, requests from an approved automated client may still contain input that an application should not process. Attackers can also send requests that resemble those of regular users or legitimate automated clients while targeting web application vulnerabilities.

Therefore, a security layer that inspects requests entering web applications therefore remains necessary as AI traffic increases.

Security Strategies for Changing Automated Traffic

Web applications now need to handle requests from a wider range of sources, including regular users, search engines, automated services, AI crawlers, and AI agents. At the same time, web attacks and malicious bot activity continue. 

Rather than relying on a single criterion or rule to evaluate all traffic, users need a defense-in-depth approach that checks different risk signals across multiple layers.

Layered Protection with Cloudbric Managed Rules

Cloudbric Managed Rules is a managed rule set designed to address a range of cyber threats in AWS WAF environments.

By applying these rule sets, users can reduce the work required to develop and continuously manage rules themselves. Cloudbric Managed Rules includes rule groups for different risk signals, including API Protection, Bot Protection, Tor IP Protection, OWASP Top 10 Protection, Malicious IP Protection, Anonymous IP Protection, and Protocol Validity Protection.

This layered approach remains important as AI-driven automated traffic increases.

For example, Bot Protection can help address behavioral patterns associated with malicious bots that affect websites and web applications through repetitive activity. Other rule sets can be applied alongside it to detect additional risks such as web attack patterns, malicious IP addresses, anonymization networks, and invalid protocol requests.

In this context, Cloudbric Managed Rules serves as one layer of a defense-in-depth strategy for addressing multiple risk factors as the range of entities accessing web applications, including AI, continues to expand.

Learn more about Cloudbric Managed Rules at Link.