Cardinal Path’s response to COVID-19 Cardinal Path is sharing all we know to help marketers during COVID-19.  Learn more.

Last week I received an interesting book with a list of the Top 500 largest retail websites ranked by annual sales. It was very interesting to see how these huge on-line retailers use SEO to generate revenue, especially the brick and mortar store that uses the web as an extension of its standard business. Usually, owners of these companies don’t focus on SEO as a revenue source and it’s quite sad to see how these online retailers miss great opportunity to increase sales by using SEO.

Here is a good example: SkyMall Inc ( is a direct marketer with about $75M in sales in 2007 and more than 450K unique visitors per month.
Let’s take a look at their SEO. Some quick research shows obvious problems.

Duplicate Content

Mirrored Subdomains

By googling a quote on their main page in quotation marks (so as to do an exact match) I found that they have two identical copies of the website under two different subdomains:


The pages in both subdomains are indexed in Google. Just use following search requests in Google to see list of indexed pages for each subdomain


This creates a duplicated content problem. It is hard to say why the website has two copies of content under different subdomains. Probably is a development copy of the live website. Anyways, all pages under subdomains should be blocked from indexing (using rules in robots.txt is the easiest way to do that) so as to avoid duplicate content problems.

Internal URL Structure

Internal URL structure creates pages with identical content under different URLs. Usually it happens when products stay under different product categories.
For example:

It’s one of typical problems for any e-commerce website. Using the canonical meta tag would help solve this problem.

Robots.txt mishaps

The Robots.txt file is the best place to block pages on website from indexing, show the location of XML sitemap, block some unwanted search engine bots, etc. Let’s take a look at

Improper subdomain blocking

There are attempts to block some subdomains from indexing


Unfortunately, these rules are not correct. Each subdomain should has its own robots.txt files with specific rules for each subdomain. For example, blocking subdomain from indexing would require file with following lines


Too many lines!

There are more than 13,000 lines of code like this sample


It’s possible to use two lines of code and pattern matching to get the same result. For example:


Linking to your XML Sitemap from Robots.txt

It makes sense to put a link to XML sitemap in your robots.txt file. For example:

  • Sitemap:

Sitemap.xml Mishaps

XML sitemap is a great way to provide search engines with a list of pages for indexing. Unfortunately Skymall’s Sitemap has some problems.

There are more about 3,000 listed URLs with internal search results. For example:

As a result Google has about 18,400 indexed pages with search results. It doesn’t make sense to provide this list of URLs with search results for indexing. It would be better to include them in the XML sitemap a list of pages with product categories and products.


Easy steps like fixing internal URL structure, using correct robots.txt files, creating useful XML sitemap, and other would make this website more spiderable for the search engines. As result, the search engines will provide better results to visitors and the website will get more organic traffic and revenue.