Below is a personal list of web spiders and web crawlers that I block from crawling various sites.
This helps mitigate issues related to server load, security, and content scraping.
Carefully consider the impact on your website’s visibility and search engine ranking before applying any of my crawler blocks within this article.
I built LinkPull because downloading a file shouldn’t require multiple steps, three tabs, and a prayer that the “Download” button isn’t actually an Ad. If you’ve ever right-clicked a link, chosen “Save As,” and watched your browser fail to save the file with the right extension — or worse, opened a new tab just to paste a URL into some sketchy downloader site — LinkPull is for you.
It’s a small, fast Brave / Chrome browser extension with one job: take a direct file link and pull it down to your computer. That’s it. No accounts, no Ads or upsells, no bloat, and the code is publicly published on GitHub for review. Like this free extension? Buy me a cup of coffee.
Organizing your website’s URL structure is crucial for both user experience and SEO. A common challenge for WordPress users, especially those using robust themes like Avada, is separating different content types.
By default, setting a global permalink structure like /blog/%postname%/ often forces all content—including FAQs and Portfolio items—into that same directory.
This guide walks you through the exact steps to configure your WordPress website so that your blog posts, FAQs, and portfolio items each have their own distinct, clean URL structures.
Moving your WordPress blog posts to a subdirectory like /blog/ is a strategic move to separate your dynamic content from static pages, improving site structure and user experience.
This article will explain to you how to achieve putting all your blog articles under a different directory.
This article serves as a reminder-to-self regarding a recent display issue I had on one of my WordPress sites with the Avada child theme.
Within the Avada theme for WordPress you can set the padding to be a negative number so that two columns or containers overlap. This generates a nice effect.
However, after updating Avada to 7.15.3 I discovered a new option under the Performance menu (see screenshot) called: Optimize Offscreen Rendering
Turning the Optimize Offscreen Rendering on prevented the overlay from appearing despite my having adjusted the Z Index options.
No doubt they’ll have it fixed in an upcoming release, for now I’ll just leave it turned off.
If your WordPress site suddenly starts dragging, your database balloons in size, or your host sends you a warning about excessive queries, there’s a good chance the culprit is hiding in plain sight: Action Scheduler.
This background task runner powers WooCommerce, subscription plugins, backups, and countless other tools. When it behaves, it’s invisible. When it breaks, it can hammer your database every few seconds and generate gigabytes of log entries before you even know what happened.
That’s exactly what happened to one of my sites — and here’s how I fixed it.
By default, WordPress saves every change you make to a post or page as a revision.
While this is great for version control, it can quickly bloat your database—especially on long-form content or frequently edited pages.
In fact too many revisions can slow down your admin dashboard, cause memory issues with certain page builders (like Avada or Elementor) as well as lead to database bloat over time.
This article is a quick reference for how I automatically limited the number of stored WordPress revisions.
In today’s digital landscape, securing your website is more than just a best practice—it’s essential. One powerful tool in your web security arsenal is HTTP Strict Transport Security (HSTS).
This policy ensures that browsers only connect to your site using secure HTTPS connections, protecting users from threats like protocol downgrade attacks and cookie hijacking.
This tutorial walks you through implementing HSTS on your website using the .htaccess file and submitting your domain to the HSTS preload list for maximum protection.
Here are three techniques to increase your website’s security posture.
This tutorial explains how to add X-Security Headers to protect against cross-site scripting (XSS), page-framing, and content-sniffing by adding extra security headers to your sites’ resources.
Hotlinking is the act of linking to a file that is hosted on another site, instead of downloading the file and hosting it on your own server. With proper source citation of course.
Images are most frequently hotlinked filetype, but zip, PDF and other digital assets can also be hotlinked.
Since hotlinking requires the website server hosting the file to use its own bandwidth to load it on the offending site, it’s considered bad etiquette. Others go further and call it theft. That’s because it eats up the bandwidth of the site you took it from without giving them the benefit of increased traffic or credit.
In this article I show you how to block others from hotlinking various filetypes from your web server. To spare your server from other undesired traffic, read my article on a list of web crawlers to block.
