Scope
This guide focuses on identity verification for automated traffic claiming to be AI crawlers, bots, or agents. It's designed for security engineers protecting web applications, APIs, and file repositories from impersonation attacks that exploit trust in self-declared identities.
You'll learn verification methods, implementation steps, and detection patterns to distinguish legitimate AI bots from malicious scanners. This guide is relevant for organizations running public-facing web servers, API gateways, or cloud storage that might be targeted by credential harvesting tools.
Key Concepts and Definitions
User-Agent Header: An HTTP header field where a client identifies itself. It's easily spoofed, so any scanner can claim to be GPTBot, Claude-Web, or another identity without proof.
Bot Verification: The process of confirming that traffic claiming to be from a specific organization actually originates from that organization's infrastructure. This typically involves checking source IP addresses against published ranges or validating cryptographic signatures.
Polymorphic Scanning: Attack patterns that vary source addresses, timing, and declared identities to evade static detection rules. The scanner observed by Cequence used more than 300 cloud addresses in a single day while cycling through approximately 32 AI bot identities.
Credential Harvesting: Automated probing for files containing secrets. The observed scanner targeted roughly 90 file paths, including .env configuration files, cloud credential files, private SSH keys, and server environment data.
Requirements Breakdown
Verification Methods
IP Range Validation: Legitimate AI companies publish their crawler IP ranges. OpenAI maintains a list of GPTBot addresses. Anthropic documents Claude-Web ranges. Your web application firewall or reverse proxy should check the source IP against these published ranges before trusting the User-Agent header.
Cryptographic Verification: Some bot operators support request signing. The bot includes a signature in the request header, and you validate it using the operator's public key. This proves the request originated from someone holding the corresponding private key.
Reverse DNS Lookup: Perform a reverse DNS lookup on the source IP and verify the domain matches the claimed organization. Then perform a forward lookup on that hostname to confirm it resolves back to the original IP. This double-check prevents DNS spoofing.
Detection Requirements
Behavioral Analysis: Monitor request patterns. Legitimate crawlers typically request HTML pages and follow links. A scanner that ignores web pages and jumps directly to .env files, .aws/credentials, or id_rsa keys isn't behaving like a legitimate bot.
Request Velocity Monitoring: Track how many requests arrive from a single identity in a given timeframe. The Cequence research observed bursts of 50 to 190 requests in roughly eight seconds before the scanner moved on. That's not typical crawler behavior.
Path Analysis: Flag any automated traffic requesting credential storage paths. No legitimate search crawler needs access to your environment configuration or SSH keys.
Implementation Guidance
Step 1: Build Your Verification Pipeline
Don't rely on User-Agent headers alone. Implement a verification layer that runs before your application processes the request:
- Extract the claimed bot identity from the User-Agent header.
- Look up the published IP ranges for that organization.
- Check if the source IP falls within those ranges.
- If verification fails, deny the request or route it to a honeypot.
Step 2: Configure Real-Time Blocking
When an identity starts requesting sensitive file paths, terminate the session immediately. Don't wait for a human review. The scanner observed by Cequence targeted files that commonly store secrets. If traffic claiming to be GPTBot requests /.env or /.aws/credentials, that's not a search crawler.
Set up rules in your web application firewall or API gateway:
- Block requests to credential file paths from unverified sources.
- Rate-limit verified bots to prevent abuse.
- Log all bot verification failures for threat intelligence.
Step 3: Secure Credential Storage
Remove secrets from web-accessible directories entirely. Configuration files containing API keys, database passwords, or cloud credentials shouldn't live in your web root. Use environment variables injected at runtime, or retrieve secrets from a centralized vault.
If you must store configuration files on disk, place them outside the web server's document root and restrict file permissions so only the application process can read them.
Step 4: Apply Zero Trust to Non-Human Identities
Extend Never Trust, Always Verify principles to AI agents and service accounts. Just because a request claims to be from a trusted AI company doesn't mean you should grant it access to sensitive resources.
Implement least privilege for automated traffic. A legitimate search crawler needs to read public HTML pages, not access administrative endpoints, user data APIs, or internal documentation.
Common Pitfalls
Trusting User-Agent Strings: The User-Agent header is self-reported and costs nothing to forge. Any scanner can declare itself to be GPTBot or Claude-Web. Always verify.
Static IP Blocklists: The attacker used more than 300 cloud addresses in a single day. By the time you've added one address to your blocklist, the scanner has moved to the next. Focus on verification and behavioral detection instead.
Allowing Bot Access to Sensitive Paths: Even if you've verified a bot's identity, that doesn't mean it should access configuration files or credential stores. Legitimate crawlers don't request those paths.
Failing to Monitor Verified Bots: Verification confirms the source, but it doesn't guarantee benign intent. A compromised bot infrastructure or a malicious insider could abuse verified credentials. Monitor all automated traffic for suspicious behavior.
Quick Reference Table
| Verification Method | Strength | Implementation Complexity | Notes |
|---|---|---|---|
| IP Range Check | Medium | Low | Requires maintaining current lists; ineffective against compromised infrastructure |
| Reverse DNS Validation | Medium | Low | Can be spoofed; use as secondary check |
| Request Signing | High | Medium | Not widely supported; requires key management |
| Behavioral Analysis | High | High | Detects novel attacks; requires baseline data |
| Path-Based Blocking | High | Low | Immediate protection for credential files |
Immediate Actions:
- Remove .env files and credential stores from web-accessible directories.
- Configure your WAF to block requests for common secret paths (/.env, /.aws/credentials, /id_rsa, /.ssh/).
- Implement IP range verification for any User-Agent claiming to be an AI bot.
- Set up alerts for automated traffic requesting non-HTML resources.
Weekly Review:
- Update published IP ranges from AI companies.
- Review bot verification failure logs.
- Analyze request patterns from verified bots.
The scanner observed by Cequence reached 48% of protected organizations that saw any AI bot traffic on a single day in October. Nearly all requests were blocked or failed, showing that verification works, but only if you implement it before the attack, not after.
