Below is a personal list of web spiders and web crawlers that I block from crawling various sites.
This helps mitigate issues related to server load, security, and content scraping.
Carefully consider the impact on your website’s visibility and search engine ranking before applying any of my crawler blocks within this article.
Are your client email Restore requests are being sent to your spam folder? Chances are their domain DNS SPF record needs to be updated.
Because Avanan inserts itself into the outbound mail flow to handle these requests, those Restore request emails originate from Avanan’s cloud IPs instead of your client’s primary mail server (like Microsoft 365 or Google Workspace).
If the client’s DNS doesn’t explicitly authorize Avanan, receiving servers will flag it as an SPF failure, thus sending the Restore request email to spam.
Two-factor authentication (2FA) is essential in helping to protect online accounts. But as more users adopt 2FA across dozens (sometimes hundreds) of accounts, the tools we rely on to manage these codes must scale with us.
Unfortunately, Microsoft Authenticator can’t fill the need for many power users. You may recall my other article on the limitations of Google Authenticator some time ago. I really thought I had something with Microsoft Authenticator, but I was wrong.
Don’t worry, the open source, privacy-first Proton Authenticator application has come to the rescue!
In today’s digital landscape, securing your website is more than just a best practice—it’s essential. One powerful tool in your web security arsenal is HTTP Strict Transport Security (HSTS).
This policy ensures that browsers only connect to your site using secure HTTPS connections, protecting users from threats like protocol downgrade attacks and cookie hijacking.
This tutorial walks you through implementing HSTS on your website using the .htaccess file and submitting your domain to the HSTS preload list for maximum protection.
Here are three techniques to increase your website’s security posture.
This tutorial explains how to add X-Security Headers to protect against cross-site scripting (XSS), page-framing, and content-sniffing by adding extra security headers to your sites’ resources.
Hotlinking is the act of linking to a file that is hosted on another site, instead of downloading the file and hosting it on your own server. With proper source citation of course.
Images are most frequently hotlinked filetype, but zip, PDF and other digital assets can also be hotlinked.
Since hotlinking requires the website server hosting the file to use its own bandwidth to load it on the offending site, it’s considered bad etiquette. Others go further and call it theft. That’s because it eats up the bandwidth of the site you took it from without giving them the benefit of increased traffic or credit.
In this article I show you how to block others from hotlinking various filetypes from your web server. To spare your server from other undesired traffic, read my article on a list of web crawlers to block.
The Facebookexternalhit/1.1 is a user agent used by Facebook to crawl and index web pages for its various services, such as Facebook, Instagram, and WhatsApp.
This crawler is responsible for retrieving content, images, and other metadata to improve Facebook’s search functionality and provide users with relevant results.
However, the facebookexternalhit crawling activity can sometimes cause issues for website owners, such as increased server load, bandwidth consumption, and potential security risks.
This article covers why you might want to block facebookexternalhit as well as how to block the facebookexternalhit crawler.
If your email account has been hacked, you must act quickly to regain control and prevent further issues.
Here are some steps you can take:
Tired of having your emails go into client junk / spam folders? You can increase the deliverability of your emails while protecting your brand and clients using free tools to create SPF, DKIM and DMARC records. With these free records in place, you can help block phishing, ransomware, and spam messages.
Email service providers like Gmail and Office 365 are increasingly sending unauthenticated emails to spam or rejecting them outright. This article will show you how to add an SPF, DKIM and DMARC record to your domain to help prevent this while keeping most malicious actors out. Note, this article is only intended for my personal use and comes with no guarantee of any kind.
However, be aware of the security risks involved with DMARC.
