List of Spider Crawlers to Block
Below is a personal list of web spiders and web crawlers that I block from crawling various sites.
This helps mitigate issues related to server load, security, and content scraping.
Carefully consider the impact on your website’s visibility and search engine ranking before applying any of my crawler blocks within this article.
WordPress Optimized Robot.txt
Before beginning, if you are running a WordPress instance for your site I advise inserting the below at the top of the robots.txt file.
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Block facebookexternalhit/1.1 Crawler
This one is particularly nasty in relation to server load and bandwidth consumption. Learn more on how to block the Facebook crawler from impacting your webserver.
Block the FacebookExternalHit Crawler via your robots.txt file using the command:
User-agent: facebookexternalhit
Disallow: /
Block Barkrowler Crawler
Block the Barkrowler Crawler via your robots.txt file using the following command:
User-agent: barkrowler
Disallow: /
Block SerpStatBot Crawler
Block the SerpStatBot Crawler via your robots.txt file using the following command:
User-agent: serpstatbot
Disallow: /
Block AI Bot Crawlers
User-agent: Amazonbot
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: FacebookBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: GoogleOther
Disallow: /
User-agent: GPTBot
Disallow: /
User-agent: Meta-ExternalAgent
Disallow: /
Blocking Other Web Crawlers
You can block all of the above listed web crawlers using via your robots.txt file using the following command:
User-agent: 360Spider
Disallow: /
User-agent: AhrefsBot
Disallow: /
User-agent: AhrefsSiteAudit
Disallow: /
User-agent: AwarioRssBot
Disallow: /
User-agent: AwarioSmartBot
Disallow: /
User-agent: Baiduspider
Disallow: /
User-agent: barkrowler
Disallow: /
User-agent: BLEXBot
Disallow: /
User-agent: Bytespider
Disallow: /
User-agent: DataForSeoBot
Disallow: /
User-agent: dotbot
Disallow: /
User-agent: facebookexternalhit
Disallow: /
User-agent: PetalBot
Disallow: /
User-agent: rogerbot
Disallow: /
User-agent: SemrushBot
Disallow: /
User-agent: SemrushBot-BA
Disallow: /
User-agent: serpstatbot
Disallow: /
User-agent: SiteAuditBot
Disallow: /
User-agent: SeznamBot
Disallow: /
User-agent: Yisouspider
Disallow: /
Complete Crawler Block list
User-agent: 360Spider
Disallow: /
User-agent: AhrefsBot
Disallow: /
User-agent: AhrefsSiteAudit
Disallow: /
User-agent: Amazonbot
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: AwarioRssBot
Disallow: /
User-agent: AwarioSmartBot
Disallow: /
User-agent: Baiduspider
Disallow: /
User-agent: barkrowler
Disallow: /
User-agent: BLEXBot
Disallow: /
User-agent: Bytespider
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: DataForSeoBot
Disallow: /
User-agent: dotbot
Disallow: /
User-agent: FacebookBot
Disallow: /
User-agent: facebookexternalhit
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: GoogleOther
Disallow: /
User-agent: GPTBot
Disallow: /
User-agent: Meta-ExternalAgent
Disallow: /
User-agent: PetalBot
Disallow: /
User-agent: rogerbot
Disallow: /
User-agent: SemrushBot
Disallow: /
User-agent: SemrushBot-BA
Disallow: /
User-agent: serpstatbot
Disallow: /
User-agent: SiteAuditBot
Disallow: /
User-agent: SeznamBot
Disallow: /
User-agent: Yisouspider
Disallow: /
Validator & Testing Tool
Test and validate your robots.txt with the TechnicalSEO.com testing tool. Check if a URL is blocked, which statement is blocking it and for which user agent. You can also check if the resources for the page (CSS, JavaScript, images) are disallowed.
Conclusion
I hope my article covering how to block various web crawlers has been helpful to you. If you have any confirmed crawlers not on this list let me know. I welcome your thoughts, questions or suggestions regarding this article.
You may support my work and future improvements by sending me a tip using your Brave browser or by sending me a one time donation using your credit card.
Let me know if you found any errors within my article or if I may further assist you by answering any additional questions you may have.
Editors Notes:
- This article was first published on Aug. 14, 2024 and was revised to include additional S.E.O. marketing crawlers and WordPress optimization.
- This article was last updated on Mar. 1, 2025 and republished on Aug. 1, 2025 with a revised list of S.E.O. marketing crawlers.
- This article was last updated on Apr. 13, 2026 to include a list of AI related crawlers.
- This article was last updated on May 8, 2026 to update and refine the lists of crawlers and AI bots.
- This article was last updated on May 24, 2026 to set “Disallow: /” after each bot as per AI tool recommendation.
- This article was last updated on Aug 8, 2026 to include SeznamBot.
