Spiderability test
From SiteRay wiki
| SiteRay Test | |
|---|---|
| First appeared: | Sitescore 3.0 |
| Applies to: | All versions |
| Type: | Individual test |
| Scored: | Always |
What does it do?
Test whether or not a website can be spidered, i.e. explored automatically by search engines and other automated programs. This is one of the most important tests of all.
Why is it important?
Spidering is the process by which a computer program (such as Sitescore, Google or Yahoo) can discover the pages in a website – it reads one webpage, looks for pages that page links to, before reading those pages, and so on. If for any reason a website cannot be spidered, it will suffer enormously:
- It can’t be explored by search engines, and will usually not appear in them (or appear lower than they should).
- It will usually be totally inaccessible, as most software to help the visually impaired or disabled is unable to penetrate the website. In many countries, this could make the website illegal.
Sites that are non-spiderable should fix this as their first (if only) SEO priority.
Example results
How is it measured?
Sitescore has a more advanced spider than most, with the unusual capability of exploring many traditionally 'non-spiderable' sites. In addition, Sitescore knows about manual pages which you have instructed it to explore through your own instruction.
This means that (paradoxically) Sitescore can usually identify non-spiderable content via use of it's own spider. Whenever a page is accessed via a means that a traditional spider would not manage, that route is marked as non-spiderable. Pages that can only be accessed via a non-spiderable route are used to calculate the volume of the site which is non-spiderable.
The score is calculated based on the proportion of spiderable pages, with a perfect 10 being awarded to sites where every page is spiderable.
Technical explanation
See the Sitescore spider.
Common problems
- My site doesn't need to be spiderable. If you're testing a site that is intentionally non-spiderable, e.g. behind a login area, this test is redundant. In Sitescore 3.1 a feature will be added to address these circumstances.
Most websites should be spiderable. If yours is not, this is a massive problem and should be addressed before all others. See the Sitescore spider.
How to improve this score
Ensure the website can be spidered. See spiderability.
How to use this test effectively
This test should return a perfect score, or very close to it. If this score is low, it should be immediately addressed as your first priority.
