During the life of a website, its quite common for pages to move around or to ‘re-structure’ your site’s files and folders. However, you can come across issues when people linking to your page go to their ’stale’ bookmark of your site, and find that page is missing – in fact, they’ll get a ‘404 Page Not Found‘. The same will go for search engines like google and yahoo, and will adversely affect your rankings within their results.
Not only do I block all accesses from libwww-perl, I also log what they were looking for which turns up an amazing amount of botnet hits on a daily basis just randomly hitting websites trying to find a way inside.
Focus: Linux, Apache 1.3+, [PHP], [MySQL]
Notes: Use at your own risk.
Summary
If you are reaching the limits of your server running Apache serving a lot of dynamic content, you can either spend thousands on new equipment or reduce bloat to increase your server capacity by anywhere from 2 to 10 times. This article concentrates on important and poorly-documented ways of increasing capacity without additional hardware.