One of the first tasks when getting to a brand new SEO project is to assess the state of on-site optimization. That is the place where we usually do the first measurement of ”how bad the things really are” or “how badly the previous SEO company screwed up” . One of the most basic tools that I always used for this task is Xenu Link Sleuth. It was great for looking at possible crawling problems, getting server headers, checking alt tags on images etc.
However, the problem with Xenu was that it is not an SEO tool. As the on-site SEO changed, so did the requirements of the indexing tool. All of a sudden, I wanted to see what pages have a canonical tag, what pages have duplicate metas issues, whether any pages have robots metatag issues and Xenu just doesn’t do this.
Enter Screaming Frog SEO Spider. If Xenu had a brother that did SEO, that’s what it would look like. It is a great tool and one of the landmarks of a great tool is constant discovery of new ways it can be useful. It has saved me numerous hours, mostly at the initial site audit stage.
I will not go into the nuts and bolts of how the tool works. The interface is very intuitive and if you have used Xenu in the past, you will feel at home with SEO Spider. Furthermore, there is a helpful video that explains how it works (British accent alert!). Instead of doing a comprehensive review of how the tool works, here are a few ways SEO Spider helped me in my SEO tasks recently:
(DISCLAIMER: SEO Spider has a free version that will index up to 500 pages. Full version – individual licence – costs £99 per year. I am paying for the full version and am not being reimbursed in any way, shape or form for writing this review):
- Investigating distribution of onsite links – When large websites want to promote a certain section of their site, one of the ways to do this is to increase the number of links to that page/section from other pages on the site. If the promoted page is found deep within the navigational structure, this will bring it closer to the root and increase the linkjuice flow to it. An overview of internal category/product pages that have a larger than usual number of on-site links pointing to them can help us identify competitor’s SEO campaign targets. Furthermore, if we cross examine these promoted pages with the number of offsite links to internal pages, we can gain some nice insights about promoted products, targeted keywords, linking strategies and other aspects of SEO campaigns.
- Identification of images without alts – this is especially true about images that serve as links. In addition to the capability of filtering images without ALTs, there are other options, as in filtering images that are larger than 100kbs or filtering images that have a long ALT tag.
- Searching link prospects for broken links you can replace – as a part of link building process, you can go to a linking prospect site, run the spider on it and get a list of all outgoing links that result in 404. Go to the linking pages, find out what was the linking purpose and the context of the broken link. Then you can either create a similar resource on one of your sites or tweak your existing content to fit the linking intention and contact the linking prospect, alerting them about the broken link and pointing out similar helpful info on your site.
- Identifying “bait and switch” links on your site – there is a possibility of webmasters creating useful content that you want to link out to. After a while, some of those websites may be taken over by other companies that want to leverage the existing links and authority and redirect them to their own resources. Thus, unwillingly, you may find your site linking to content that you did not intend to link to. In the mild case, your visitors will click through and will not find the useful content you intended them to. In worse cases your site may be flagged as linking to adult and maybe even illegal content. Hence you can get a list of all the external links that result in a redirect and check them out. A lot of the cases those will be simple canonicalization issues that webmaster took care of (adding the trailing slash, pointing to the www version of the site, reflecting a change in directory structure, etc.)
- Discover noindex/nofollow metas – A great way to get an overview and identify screw-ups left behind by your developers. Leaving noindex metas sounds like a beginner mistake, but you would be surprised how many experienced SEOs stumble here, either due to their own oversight or due to forgetful developers
- List all the canonicals – canonical tags are a great way to get rid of duplicate content issues on site (and across sites) but it can be used for other, more sinister purposes too (that’s for another post). Conveniently, SEO Spider will spit out all the canonicals across the site
- Search inside the HTML source of pages – this is one of the recent additions to SEO Spider. It can be very useful if you want to find all the pages linking to a target page with a specific anchor text or look for tracking code within pages. Up to 5 custom searches can be defined in the Configuration section.
These are just a few things I have found in the last few weeks. I am sure this list will just grow, so check it out at later stage, I may be adding more great ways to use SEO Spider