Your first job during an SEO campaign is to ensure search engines can find and crawl all of the content on your site.
Search engines index the web in a series of steps:
- The search engine sends out a ‘crawler’ (also called a ‘spider’ or a ‘bot’, depending on your level of nerdiness). The crawler jumps from page to page on a given site, following the links in each page to the next.
- The crawler transfers the content of the page back to the search engine.
- The search engine indexes each page, counting the number of times a word or phrase occurs on the page. In doing so, it determines relevance.
- And (this is the fancy part) the search engine scores the authority of each page by counting and scoring incoming links from within the same site, and from other sites. These links also help determine relevance.
- The search engine will also bring in a huge collection of other factors, many of which we can only guess at, to further determine relevance and authority
But notice – it all started with the crawler. If you’re gonna get ranked, you gotta get found, first. This is especially true in long tail search. And, as you’ve heard me say 10000 times by now, the long tail is where it’s at.
That’s why an entire section of this training focuses on site visibility. Of all of the sub-sections, the fifth, Diagnosing roadblocks, is the most important. But the first four provide you with a lot of great background on how to use various tools to find problems. They also show you how to use feeds and sitemaps to make sure your site gets crawled, and how to use some of the cooler crawling tools out there to help with the whole process.
Read on:
- Crawling your site: Tools and techniques
- Feeds and sitemaps
- NERD STUFF: Log analysis
- NERD STUFF: Getting real (time)
- Diagnosing roadblocks