It’s not easy explaining redirects without losing your audience, as header status codes and redirects are not the most compelling subjects. To avoid those terrible scenarios, most SEOs simply tell their clients, “301 redirects are good, and 302 redirects are bad.” 99.99% of the time, that’s all that anyone needs to know. And regardless of whether or not anyone really knows the reasoning behind that claim, the fact is that everything almost always works better for SEO by using 301 redirects.

But I say we can’t stop there. Why restrict the boring nature of a redirects to verbal conversations only? I say we must take this exhausting topic to the blog! We need to have it down in writing! We must dive deeper into the topic of redirects! That’s what I say. So without any further ado, I’d like to get started on this long-awaited, long-winded topic: HTTP Status Codes, specifically the 3xx series. Here is a brief summary of the 8 HTTP status codes from the 3xx series:

• 300 Multiple Choices
  Indicates multiple options for the resource that the client may follow. It, for instance, could be used to present different format options for video, list files with different extensions, or word sense disambiguation.

• 301 Moved Permanently
  This and all future requests should be directed to the given URI.

• 302 Found (aka Object Moved aka Moved Temporarily)
  This is the most popular redirect code, but also an example of industrial practice contradicting the standard. HTTP/1.0 specification (RFC 1945 ) required the client to perform a temporary redirect (the original describing phrase was “Moved Temporarily”), but popular browsers implemented it as a 303 See Other. Therefore, HTTP/1.1 added status codes 303 and 307 to disambiguate between the two behaviors. However, the majority of Web applications and frameworks still use the 302 status code as if it were the 303.

• 303 See Other
  The response to the request can be found under another URI using a GET method. When received in response to a PUT, it should be assumed that the server has received the data and the redirect should be issued with a separate GET message.

• 304 Not Modified
  Indicates the resource has not been modified since last requested. Typically, the HTTP client provides a header like the If-Modified-Since header to provide a time against which to compare. Utilizing this saves bandwidth and reprocessing on both the server and client, as only the header data must be sent and received in comparison to the entirety of the page being re-processed by the server, then resent using more bandwidth of the server and client.

• 305 Use Proxy
  Many HTTP clients (such as Mozilla[4] and Internet Explorer) do not correctly handle responses with this status code, primarily for security reasons.

• 306 Switch Proxy
  No longer used.

• 307 Temporary Redirect
  In this occasion, the request should be repeated with another URI, but future requests can still use the original URI. In contrast to 303, the request method should not be changed when reissuing the original request. For instance, a POST request must be repeated using another POST request.

How about that? I bet you didn’t know that there were actually eight unique status codes for redirects. If you read through that list, you may have noticed that, technically, the 307 is the Temporary Redirect. The 302 is actually listed as Found or Moved. Furthermore, they had to create the 303 and 307 responses after various browsers were interpreting the 302 differently. Fascinating, right? I wonder how many SEO’s actually know about the subtle differences between a 302 and a 307.

But let’s not stop there. Apparently, it looks like the 303 is actually what most SEOs talk about when they speak of a 302 temporary redirect. Ian McAnerin describes it very well on his blog, so I borrowed this next part from him:

In reality, a 302 just means “Object Moved”, or “Found”, which, if accompanied by a target URL, browsers and servers interpret as a redirect. But with no target URL, they will happily stay there, and it’s not an error.

In reality, a 303 is what most SEO’s think a 302 is. A 303 means “See Other”. A 307 is the actual Temporary Redirect. It really means temporary, as in the very next request should also be made to the old URL, and the new one should not even be cached. This is usually only used for emergency redirects (like when a primary server is down) and the like.

A302 doesn’t dictate a redirect, it just says that what you were looking for moved, and it’s been found there. You are usually redirected only as a courtesy and for usability purposes. Technically, you should use a 303, which really does the job properly.

How about that? Can you believe it? Have you even heard of a 303 redirect? It’s crazy, huh?

If you’re still with me, I know this information is probably news to you. Or your eyes have glazed over with the inspirational information in this post. I mean, this stuff is obviously rocket science. At any rate, the good news is that you don’t really need to have an in-depth knowledge of these redirects (well, unless you run into someone who might dismiss you because you don’t know this stuff) because it’s true that 301 redirects do take care of everything 99.99% of the time. The other good news is that now you know about all of the redirect status codes. Now go forth and 303. Er, uh, I mean 301. Or was it 307? :)

When will Google remove those old URLs from their index?

Hey Yahoo! Have you even heard of a 301 redirect?

Partial Credit for Bing. They never really indexed our old blog very well anyways.

All 3 top search engines still have a lot of URLs indexed from the previous blog location:

• Google: 127 URLs indexed
• Yahoo: 244 URLs indexed
• MSN Bing: 29 URLs indexed

This is not surprising in the least bit. While all search engines advocate the use of 301 redirects when changing URL structures and/or domains, they don’t seem to remove the old site URLs very quickly. In my experience, it can take a up to 3 months. It really just depends on the number of URLs and other factors. I mean, maybe googlebot is having a rough day or something.

The old Range blog at rangeonlinemedia.com/blog had about 300 URLs. Included in those 300 URLs were post pages, archive pages, tags, author pages and other links on the blog. As mentioned earlier, we created 301 redirects for all URLs. Here is how we redirected those URLs:

• Posts (301 redirected on a 1-to-1 basis to the post URLs on the new blog)
• Archives (301 redirected all archive pages to the http://therangeblog.com/)
• Tags (301 redirected all tag pages to the http://therangeblog.com/)
• Author pages (301 redirected all author pages to the http://therangeblog.com/)
• Images, CSS files, RSS links, etc… (301 redirected all misc file URLs to the http://therangeblog.com/)

I typically advocate 301 redirecting every URL on a 1-to-1 basis to its counterpart URL on a new site. For TheRangeBlog.com, we did that for post pages. For all other URLs, we created a wildcard redirect. Basically, all non-post URLs are 301′d to the homepage of TheRangeBlog.com. It was a time/resource issue. I know, I know. Not perfect, but it’ll have to do. BTW, none of our old blog URLs had any links, so I wasn’t too worried about 301′ing a random tag URL to its counterpart on the new blog. Just sayin’ is all…

Well, that’s about it for now. Stay tuned for more – in the future!

• We removed the placeholder HTML file for the homepage.
• We removed the disallow entry from the robots.txt file.
• We updated the privacy setting in Wordpress to allow search engine access
• We updated our corporate website to reflect the new blog URL.
• We created and uploaded the 301 redirects from all of the old blog URLs to the counterpart URLs on the new blog domain. [Note: We did not write 1-to-1 redirects for every URL. Due to time constraints, we only created 1-to-1 redirects for the 60 or so blog post URLs. For all other URLs, we created a wildcard redirect that 301'd all URLs to the new blog's homepage. Due to our previous blog's lack of links, I caved on this item. In most other cases, I recommend 1-t0-1 redirects when changing URL structure or moving from one domain to another.]
• We built an XML sitemap for the new blog.
• We created/verified Webmaster accounts with Google, Yahoo and Bing. In these accounts, we submitted our XML sitemap location. All sitemaps were accessed within 24 hours of submission.

All of that took place last Tuesday (3-Nov-2009). Today is November 10, 2009. How have the search engines responded? Let’s take a look at our site counts in Google, Yahoo and MSN Bing:

Google Site Count: 8 URLs indexed

Yahoo Site Count: 1 URL Indexed

MSN Bing Site Count: 49 URLs Indexed

Just in case you cannot see the pictures, here are the results:

• Google: 8 URLs indexed
• Yahoo: 1 URL indexed
• MSN: 49 URLs indexed

That’s amazing to me. In my experience with crawling speed and indexing speed, Google typically wins big. But after a week of TheRangeBlog.com being live and active, Bing is clearly winning the race to index our blog. Nice work, Bing! Thanks for taking note of our blog.

I should also note that our blog has almost no inbounds links. Aside from the links from our corporate site and from a few Range Twitter accounts, we have virtually no link authority. Our lack of inbound link love is obviously playing a role in how quickly our site is getting indexed. But worry not – because we are working on links! (And if you are the linking kind, please consider linking to us. I won’t beg, but I would appreciate any link juice. Well, not from bad neighborhoods. Yes, I am an SEO.)

To make your way through this future of shiny metal and emotionless automatons, an important but under-appreciated page on a website is the “robots.txt” page. It serves as a guide for robots on how to behave on a site. Other robots besides search engines, including some nefarious robots (kind of like Decepticons) can sometimes be managed by this file as well.

Some tips for ensuring your site is prepared for search engine robots include:

• List all of your XML sitemaps. An XML sitemap is an easy source for robots to find all of your pages, videos, and other files. By including the sitemap URLs here, you are ensuring search engines can send traffic to any of your pages.
• Tell the robots which areas are off-limits. This doesn’t involve Asimov’s Three Laws, but you can prohibit access to files and directories that shouldn’t be found in search engines. Additionally, you can keep robots focused on crawling only important pages and keep them from spending all their time on unimportant content.
• Always serve the same content to robots as users. Otherwise you can be hit with a penalty for misleading the search engines.
• Avoid crawl delay for search engines. In rare cases, a robot can take up a lot of your server’s resources. You can add code to slow the robot down in these cases, but slowing a major search engine down can cause a traffic loss.
• Test your file! Once you’re all done, use the free tools from Google Webmaster Tools and Bing Webmaster Center to make sure everything is working properly.

This page can always be found at www.YourGreatSite.com/robots.txt. Don’t be surprised if it hasn’t been created – even some major sites like Yahoo haven’t made one.

• Keyword: (random 7-term keyword)
• Impressions: 1
• Clicks: 1
• Cost: $.25
• Conversions: 1
• Revenue: $1,256.37
• ROI: Hahahahahaha

It may be the only impression that particular keyword gets for the next 6 months. But if you can create a campaign that reaches out effectively enough to reach customers looking for very specific things, these one-click wonders can easily leverage investment in wider ranges of nonbrand keywords – i.e. make you look super smart.

With a solid SEO program and smart keyword selection, you can accomplish the same thing – only exponentially larger, given the volume of clicks that natural results drive versus paid ones. All it takes is a readable site and the time, patience and courage to see it through (though having an award-winning SEO department at your back can really help), via opportunities like:

• Taking your meta data beyond single-word targets based on what you’ve named your categories or products
• Taking your product names and descriptions beyond what your branding team or manufacturers provide you
• Taking your internal linking beyond a three-clicks-to-a-product mentality and finding ways to get engines from your most authoritative pages to your most valuable content faster

Combining these efforts will open up your site to the ability to rank for a much wider range of keywords, in much the same way that a six-digit keyword buildout can, with much the same effect – only without those pesky click costs!

Well, don’t take your Nikes off just yet. Google recently announced a slate of updates and new developments, including ways to make your search results prettier and a new emphasis on fresh content. For those who’ve been encouraging clients to add microformats to their pages, you should feel a little vindicated, because these should be right up your alley. If that’s not enough, MSN also announced that they’re updating and rebranding Live Search, naming the concept Bing (previously known as Kumo) and gearing it towards behavioral search. If it works like they say it will, then we’ll have something to celebrate before July 4.

So that’s some good news for the summer, but it’s more applicable for the winter. In terms of development, it’s time to start work (or at least planning) on site enhancements for the 2009 holiday season!

‘But it’s not even June!’ you might be thinking. Just remember to take into account a few important factors:

• Your Site Will be Untouchable in 4 -5 Months: Typically, large ecommerce sites go into a code freeze in October or early November at the absolute latest. A code freeze is a period of time where developers can work on site enhancements, but can’t actually push anything live for anything less than a life-or-death emergency in order to make sure that the site isn’t knocked out by errors at a critical time of year. So, in reality, it’s more like August rather than June on your production schedule.
• Development Takes Time: The planning, pre-development legwork, development and eventual launch of changes to your site – SEO or otherwise – is a long process. Depending on your team and your process, you’re looking from two to six months just to get even simple, natural search-affecting changes made to the site. If you try to start making massive changes to your site in August, you could very easily wind up seeing your Holiday 2009 ambitions coming up sometime next March without some serious overtime or help.
• Search Engines Take Time: If SEO gains are even part of your goal, once you’ve made improvements to your site that are meant to improve your natural search results and seen them pushed live, you’re at the mercy of the engines. It can take anywhere from two days to four weeks to get your changes indexed, and another six to twelve weeks of fluctuations as Google, Yahoo and MSN fully digest your changes and start really advancing your positions.

If you start counting backwards from December with these in mind, Santa’s shadow looms pretty large.

Now is the time to start making your list and checking it twice. Start by looking at your holiday keyword and content strategy, building out your gift pages, fixing any leaks in your natural search potential and getting your site ready to rank for potentially the toughest holiday season in decades. Be a good boy / girl now, and you could find yourself with the kinds of gifts we’ve all been asking for all year: Revenue, growth, and a shot at Holiday 2010.

Because of its mysterious nature, people draw a lot of false conclusions about how a web page can rank. For one, there is no magic bullet to ranking. In fact, Google has over a hundred data points it factors into rankings. They usually deliver good results, too, which keep getting better. What that means is that you should focus as much on creating what a visitor would like to see highly ranked, as well as working on SEO tactics.

Some other common misconceptions are:

• You can’t have a flash(y) site and rank well. Not so, in fact, you can have a great looking site with graphics and even Flash movies widely used with careful planning. Using CSS techniques and font alternatives you can remove many unnecessary images. Those images that are kept can be thoroughly optimized. Flash movies can also be better optimized and integrated. With that in mind though, stick with text first and add in these elements when they add to the user experience.
• Traffic is the measure of an SEO campaign. While it’s true that a good SEO campaign will drive more traffic, the traffic needs to be qualified. Search phrases with low conversion rates, low bounce rates, and low visitor loyalty rates should be lower priorities than well performing phrases. Additionally, landing pages need to be well designed for conversions as well as capturing search traffic.
• A lot of sites assumed an XML sitemap wasn’t important, so long as you have an HTML sitemap on your site. In fact, Google discovers new pages on a site much faster when an XML sitemap is present and updated frequently. That means your new products start getting traffic quicker.
• Ranking well is your goal. Yes, ranking highly in Yahoo is nice, but is anyone clicking on your listing? Search visitors won’t simply click on the first listing, they will quickly scan some listing before clicking. Your title, description, and URL need to convince a user to click.
• SEO just affects your site pages. While the HTML pages of a site are a huge part of SEO, pages are not the only content appearing in search engine results. Videos, images, PDFs, RSS feeds, Flash movies, and more appear sprinkled throughout search engine results. Including elements like these in your site, and ensuring they are optimized, means more opportunities for your site to appear.

What this post is about, however, is the mindset that drives this question: designers and site owners all want a slick interface for their site, built with universal usability for any and all visitors. Technically, this usability is supposed to include search engines, but it’s typically included in a perfunctory way—it’s sort of like sending a wedding invitation to that cousin you never see and barely know. On filigreed cardstock, you really want them to show up, but not as much as say, the family friend who likes to give envelopes full of money to newlyweds. As if that weren’t bad enough, for many site designers, search engine accommodation is a detail they care so little about that they’ll leave it to the search engines and users to bend to them, rather than the other way around.

In Range’s recent Search Marketing Now webinar on SEO and Usability, we had a chance to discuss the ways that your site’s usability and natural search potential actually walk hand-in-hand. For anyone who couldn’t attend or can’t listen in to the on-demand version the main takeaway is this: your natural search potential and conversion potential are, in many ways, the same thing. They’re often controlled and helped – or hindered – by the same elements, so don’t be willing to trade one for another.

As we all know, the online world’s rate of evolution increases every day, and there is a certain panicky pressure to be the first to jump on or keep up with the latest technology and techniques. For example, first there were JavaScript tricks, which were later reinvented for Flash and then most recently for AJAX. Each iteration made entirely new web experiences possible while still providing ways to make the same content accessible and clean to standard browsers, screen readers, mobile devices and, of course, search engines. And most new web tools are like that. But nearly every time, these opportunities get ignored in favor of even more usability-improving technology just not worrying about it, at least un until someone notices that traffic and conversion aren’t what they could be. Then you’re suddenly rushing to build fixes into the system that should have been there in the first place.

Now I’m not saying that you shouldn’t use Flash, or AJAX, or giant navigational images. It doesn’t take much to work with these elements to set yourself up for SEO success, but be aware that the more advanced the technology, the less it takes to doom your site to a comfortable spot on the fifth page of Google results. So what do you do? The solution is a pretty simple paradigm: build your site architecture with all of your users in mind, but ESPECIALLY SEARCH ENGINES. You’ll save yourself a lot of hassles in the long run.
In other words, the next time you’re designing a site, try something crazy: go back to the basics. Plan and design for the least common denominators and construct the simplest but most necessary elements in your site first. Once you’ve taken care of the basics essentials, then you can look for ways to layer in the bells and whistles everyone’s drooling over – as long as those basic elements aren’t affected.

This is how you build for SEO and for usability without having to go through costly and time-sucking changes later. It’s also the way to wire your house for electricity without having to tear out drywall: Do it before the drywall goes up!
To sum:

1. Your SEO and usability are affected by the same things – don’t sacrifice one for another.
2. Designing and building with all your users in mind is the way to thread this particular needle.

And no, Flash still isn’t readable, but you’ll hear it here first if that ever happens.