|
Custom 404 Page Kills
Spiders
Let's talk about spiders for a second.
You know, the same technology that the search engines use to index your
pages. When's the last time you ran one on your own site? Did you know
that your custom "404 not found" error page could be driving them away?
You'll want to sic a spider on your own site, before letting MSN, Google
or Yahoo get at it. Why? To make sure its spiderable, that's why! Huh???
Didn't think about validating your linking structure, did you?
There are many page whackers available that download a whole site to
your hard drive, but who knows what kind of "forgiving" spidering
technology they're using. The best one, the one that replicates the
crawling spiders like Google and Yahoo the closest, is a tool like
OptiSpider.
Here's a tip, if OptiSpider can't spider your site... neither can
Google. That's right, it's probably just about the best reason to get
and use a tool like OptiSpider, just so you can see the same stuff the
spiders see.
If you manage multiple sites for clients, it's a great weapon in your
toolbox! I can't count how many times I've done coaching sessions for
clients who claim search engines are not finding all their pages. A few
seconds later - using OptiSpider - I can tell them... dude!!! You didn't
link to the pages properly, or your custom 404 not found page is killing
the search engine spiders.
No Google, Yahoo or MSN spider is going to find all your pages, when you
put the crawler in an endless feedback loop. You're lucky if the spiders
ever come back.
Stop doing whatever it is (even if it's reading this article ;-) and go
look at your custom 404 page. Everyone loves to use them. Unfortunately,
many people use them badly. The biggest sin of all - even I was guilty
of this - is using a relative link on the 404 page. It's so common that
relative linking should be banned altogether.
Imagine the spider is several directories down in your site and
encounters a missing page or broken link, up comes your custom 404 error
page. Too bad sucker! You used a relative link instead of an absolute
one. The spider is totally miffed at your site and resets, leaving all
the rest of your pages behind... unspidered, unindexed and unfound.
Stop whatever it is you're doing and go do it now. Put an absolute link
on your 404 page, one that includes the http://www. and your full .com
web address to your home page. That way, when your custom 404 error page
comes up, the spider won't choke. Instead, it finds your home page from
the custom 404 page and continues spidering like normal.
If you want to make spiders happy and be sure they can crawl your entire
site, then use a spidering tool like
OptiSpider to test your internal linking structure. Just don't be
surprised if you end up getting a whole lot more pages indexed by the
search engines. What's on your 404 page?
by
Michael Campbell
Author of....
Revenge of the Mininet...
Advanced search engine linking
strategies and diagrams for increased revenue.
Clickin' it Rich... The complete
work from home business training
system for new affiliates.
Nothing but 'Net... Simple
internet marketing
strategy that made $750,000 in less than a year.
Internet Marketing Secrets Newsletter...
Learn how to harness the money making power of multiple internet revenue
streams like search engines, affiliate programs, paid advertising,
opt-in email, newsletters, ebooks and lot more in
internet marketing
secrets. Subscribe now and get a free lifetime membership. |
