The Facebookexternalhit/1.1 is a user agent used by Facebook to crawl and index web pages for its various services, such as Facebook, Instagram, and WhatsApp.
This crawler is responsible for retrieving content, images, and other metadata to improve Facebook’s search functionality and provide users with relevant results.
However, the facebookexternalhit crawling activity can sometimes cause issues for website owners, such as increased server load, bandwidth consumption, and potential security risks.
This article covers why you might want to block facebookexternalhit as well as how to block the facebookexternalhit crawler.
There are several reasons why you might want to block the facebookexternalhit crawler:
- Server Load and Bandwidth Consumption: Facebook’s crawler can generate a large number of requests, which can put a significant load on your server and consume bandwidth.
- Security Risks: Facebook’s crawler can potentially expose your website to security risks, such as SQL injection attacks or cross-site scripting (XSS) vulnerabilities.
- Content Scraping: Facebook’s crawler can scrape content from your website without permission, which can lead to copyright infringement and other legal issues.
- Indexing and Caching: Facebook’s crawler can index and cache your website’s content, which can lead to issues with search engine optimization (SEO) and content freshness.
To block the facebookexternalhit crawler, you can use various methods, including:
- .htaccess File: You can add the following code snippet to the end of your Apache web server .htaccess file to block the facebookexternalhit crawler. Note, this will block all requests from Facebookexternalhit/1.1:
- IP Blocking: You can block Facebook’s IP addresses using your server’s firewall or IP blocking software. However, this method is not recommended as Facebook’s IP addresses are constantly changing.
- User Agent Blocking: You can block facebookexternalhit using user agent blocking plugins or software, such as Apache’s SetEnvIf directive.
- Cloudflare: If you are using Cloudflare, you can block facebookexternalhit using Cloudflare’s firewall rules.
Important Notes
- Blocking facebookexternalhit/1.1 will impact Facebook’s ability to crawl and index your website, which can affect your website’s visibility and search engine ranking.
- Facebook’s crawler is designed to respect the X-Robots-Tag header, which can be used to specify crawling and indexing instructions.
- Facebook’s crawler is also designed to respect robots.txt files and will not crawl pages that are disallowed by robots.txt. Example robots file:
User-agent: *
User-agent: facebookexternalhit/1.1
Disallow: /
SITEMAP: https://website.com/sitemap.xml
Pros: Blocking the facebookexternalhit crawler will help mitigate issues related to server load, security risks, content scraping, and indexing.
Cons: However, carefully consider the potential impact on your website’s visibility and search engine ranking before blocking the facebookexternalhit crawler.
I hope my article blocking the Facebook Crawler facebookexternalhit/1.1 has been helpful to you. I welcome your thoughts, questions or suggestions regarding this article.
You may support my work and future improvements by sending me a tip using your Brave browser or by sending me a one time donation using your credit card.
Let me know if you found any errors within my article or if I may further assist you by answering any additional questions you may have.
A few more words about the effects of blocking the FB robot and possible temporary action.
The blocking of images on FB from blogs/other websites does not apply to images published directly on FB – these are visible all the time.
Temporary solution for those publishing blog entries: of course you can publish on FB not only a link to the entry from which FB will download the title image, but also the title image itself.
Thank you for taking the time to provide the additional information!
I have followed your ‘how to block facebookexternalhit’ on https://endurtech.com/block-facebook-crawler-facebookexternalhit/ and added the code in the .htaccess file, but facebook keeps hitting (attacking) my website, around or almost 10GB per day in traffic, do you have any other suggestions I could try?! My host already has suspended me twice because of bandwith usage. And as expected Facebook does not respond (mailed multiple times to webmasters@meta.com and facebookbot@fb.com)…
Sorry to hear Facebook is still hitting your site. Two other solutions may include blocking the Facebook crawler via the robots.txt file or blocking the Facebook crawlers’ IP range (not recommended). If you use the robots.txt solution here is the code:
User-agent: facebookexternalhit
Disallow: /
Thanks for this article. Twice now I’ve been caught out by this crawler and the only efficient way to stop it is to block it completely. How can Facebook justify this behaviour?
Happy to have helped Pixolomew. Facebook wants your data, they’ll do what ever it takes to get it.