Designing SEO Experiments – a different angle

January 26, 2008 – 11:06 pm

I have thoroughly enjoyed SlightlyShadySEO‘s post about designing SEO experiments. If you haven’t read it yet, go do that now, it gives quite a comprehensive list of all the variables needed to be controlled when creating a sterile environment necessary for isolating and testing the individual ranking algorithm parameters. I felt that there are few points worth expanding a bit:

Creating sterile environments is really important in science, since one always wants to look at the influence of changing a single parameter, while keeping all the others constant. The same premise is valid for constructing experiments in search engines. However, there are at least two drawbacks in this translation of scientific principles into the SEO world:

  • It is very hard to create a sterile environment. Even if you create a number of websites, all ranking for the same keyphrase and then start changing one of them, the complexity and the hidden nature of SE ranking algorithm always presents us with dilemmas regarding the cause and effect of the change we observe.Let’s say we have increased a keyphrase density of one of the websites and it drops in rankings. Is it because we crossed some threshold value after which increase in density starts invoking penalty ? Or is it because someone scraped the site, thus inducing duplicate content issues ? Or maybe there was a change in the ranking algorithm which made keyphrase density less important ? There are innumerable possibilities for things that could have affected our SERPs and it is very hard to point at one of those as the primary contributor to the change.
  • Conclusions reached in such sterile environments are not always valid in competitive niches. For example, you want to test the effect of adding irrelevant incoming links. You create a number of links to your testing site from unrelated pages and with varying, irrelevant anchor text. After two days, your site experiences a huge surge in locations in your sterile environment. Those locations stick for a few months and you conclude that getting links improves your locations, regardless of their relevance to your topic or keyphrase you are testing for. You then do the same to a site in a competitive niche. The locations increase after a few weeks, but then start to drop, until your site lands only 5 positions above its starting point. You are baffled, since it is not what has happened in your sterile environment

The main reason for these differences is that you are seeing only a very crude presentation of possibly subtle differences between sites – the SERPs. In other words, there is this big, complex machine – the ranking algo – which adds and subtracts points from a website’s score, according to different parameters that the algo takes into consideration.

Imagine the final ranking score being a bucket of water to which liquid (the points) is constantly being added and taken out of by different glasses, cups, spoons and straws. You do not see how much each of those vessels has added or taken out of any particular bucket. Hell, you don’t even see how much liquid there is in each bucket at any given time. You only see that final order of buckets, and you know that they are ordered by decreasing amount of liquid in them.

If we assume that the final ranking of sites represents a decreasing order of scores assigned to those sites, another mystery is the actual gap between the scores. Top 5 locations on Google could represent 5 point differences between each of those sites:

  1. Site A – 500 points
  2. Site B – 495 points
  3. Site C – 490 points
  4. etc.

If, on the other hand, there is a 100 point gap between each of the sites, we would get a different scoring scale:

  1. Site A – 500 points
  2. Site B – 400 points
  3. Site C – 300 points
  4. etc.

Obviously, an increase of 20 points in one of the sites’ ranking score would have a dramatic effect in the first SERP and absolutely no effect in the second. It also makes sense that “sterile” environments, where all the websites are under the tester’s control, look more like the 5-point-gap SERPs than the 100-point-gap model, thus creating a difficulty in reaching conclusions applicable on competitive SERPs by experimenting on the artificially created ones.

So, in spite of these issues, how do we design experiments and still reach valid conclusions relevant for competitive niches ?

In SEO, like in real world science, the key to this problem is repetition. One can conduct experiments on competitive niches, but single experiments will not teach us anything about effects our actions have on SERPs. It could be that a certain link filter has just kicked in or that a particularly valuable link was taken off. However if a phenomena repeats itself over a number of niches and/or in the same niches several times, the chances of your action being the main reason for the observed change, get just a little bit closer to certainty.

Tags: , , , , ,

  1. 4 Responses to “Designing SEO Experiments – a different angle”

  2. Very interesting posts, but personally, I think you are better off working on building incoming links than carrying out SEO experiments.

    By Dave Toys on May 4, 2008

  3. Interesting post.

    I’m not so sure (in reference to the above comment). I think you could learn some important information from conducting SEO experiments. People are always saying that H1 tags are treated as more important than other html body tags. However, this is going on blind faith. There probably aren’t many people who have actually ran experiments on this to see what difference they can really make.

    By Costas on May 7, 2008

  4. ,

    By name on Jul 14, 2008

  5. @ Dave Toys – every day in SEO is a learning day, link building however important it is, is only a smidgen of the important ranking factors that combine to create an authority website, experimenting is one of the best ways an SEO can exceed his competitors… theres no point in just doing what everyone else does is there!

    By Stuart Chester on Mar 1, 2011

Sorry, comments for this entry are closed at this time.