There are a number of common ‘build’ obstacles that can affect how easily your site can be crawled. Fixing these hitches is a fundamental part of on-page optimization and can make a real difference to the volume of traffic you receive. Here are a few bug bears:

Images with no alt text: Search engines cannot read images. Your page may look beautiful thanks to the glorious image you’ve included but it will be completely invisible to the search engine’s spiders. Likewise if you have submission forms, diagrams or text box images, none of the important content you’ve created will get its due credit from the likes of Google etc. This is easily remedied with either an alt tag providing a description of the image that the human users can see, or by ensuring your text is not embedded in an image file, but as HTML text.

Flash heavy and framed sites: As with images, spiders don’t deal very well with Flash or Java. While Flash can enhance a user’s experience of a site, as long as it doesn’t slow down the loading time, it hides important content from the spiders. Highly creative sites commonly have their homepage in Flash, which looks great but is detrimental to the site given that most inbound links will point there, yet the spiders can’t see it. Finding ways to secure a balance or at least ensure there is a text option available is crucial to a successful ranking. Similarly, a framed site can cause spidering problems as frames tend to fragment your site. Content from pages within the overall framework can be hidden as the spiders might only read one of the frames on a page. It is best to avoid building your site in frames if possible.

General functionality: Being too clever, including whizzy graphics and such like can sometimes slow down your site. The user experience has to be as smooth and as uninterrupted as possible so avoid over-complicated functions, ensure file sizes are low for downloads etc and invest in some high quality server hosting. Checking your site works in different browsers is prudent and it also pays to check regularly for broken links in order to limit user frustrations and also ensure that all content can be accessed by the spiders at all times. Having a site map helps both user and spider as, if your site’s not vast, it means that most of the content can be accessed in two clicks.

Duplicate content: Search engine spiders like unique content. If they find duplicated versions of the same content, they will take one page as ‘canonical’ and display that single page as the result. This is particularly damaging to your ranking in the search engine results as the strength you have built up through the links to a particular page will be diluted by the additional copies. Commonly this can occur on a site’s homepage when multiple versions of the URL are used e.g. the inclusion and exclusion of the ‘www.’. As most external links tend to point to the homepage, it is important to protect its strength which can be done simply by using a 301 redirect. This ensures you dictate the canonical page to the search engine spiders by pointing all duplicate pages to the primary page.

Design and navigation: The design and navigation of your site can guide a user seamlessly through your content on a path you have pre-ordained. This is the same for the search bots. By creating a clear and logical network of internal links, obvious navigation systems, breadcrumbs, well written anchor text so it is clear what will be returned on the click of a link, and so forth, you not only create a smooth user experience but lead the bots deeper into your site. Search engines look for relevancy so designing your navigation system to act as a funnel; taking them from broader to narrower subject matter within an overall theme, will help them rate your relevancy and suggest there is depth and quality to your site.

These are just a few of the most common adjustments that can help improve your site’s crawlability. Speaking very generically, it stands to reason that the better the experience for the web user, the better it will be for the search engines. The key is to make sure a literal, textual translation is provided at all times for the internet’s creepy crawlies.

Read more about increasing visibility and crawlability in the search engine’s through the optimization of your keywords.

Start Analysis

In the news

BMW got banned from the Google search results due to their use of doorway pages. At the time, their Javascript site was unsearchable by Google’s spiders, so they created ‘doorway’ pages that focused on ‘used car’ keywords and redirected the user to the BMW main site. This inflated their position in the SERPs – until Google found out. While BMW denied misleading web users, on the grounds that the content users were redirected to wasn’t any different in principle to the doorway page, they were still penalised with a zero PageRank as Google maintained that all doorway pages were illegal no matter what. (Original article: BBC News, Feb 6 2006, BMW given Google Death Penalty). It is not fully documented as to the ultimate cost this had to BMW but is probably sufficient to say that the volume of lost traffic and converting customers must have dented their bottom line somewhat.

Don't let sloppy SEO catch you out

It’s not always instantly apparent that you have been banned. One way to check whether your site has been excluded or not is to type ’site:www.yoursite.com’ into the Google toolbar. If you don’t turn up, it would suggest you’re no longer in their index. There could be some perfectly innocent and easily solvable explanation for this. It could be that your Robots.txt file that controls what the Google bots can and cannot crawl is set up incorrectly. Instead of blocking them from a certain area, you may have blocked them altogether, thus preventing them from indexing you. You might even have your meta tags wrongly coded and be unwittingly bouncing the spiders away as soon as they get to you. These mistakes are easily rectified and will result in you being re-indexed once fixed.

Black hat bans
It is possible that you may have fallen into some shady areas of SEO practice by mistake. For example, if you set up domains for .com, .co.uk, .org and so on to protect your brand or business, but duplicated the content on each domain, Google will punish this. Choosing a primary domain and putting a 301 redirect from the other sites is the best way to get out of hot water in this case.

You might also want to double check what sites you’ve linked to. If you’ve been doing things properly you’re likely to know this anyway, but giving the innocent the benefit of the doubt, it is possible to link to a site that is not behaving ethically without realising. You will get tarred with the same brush once their bad habits are found out and pleading ignorance may not get you out of it.

Getting re-indexed

If you do find out you’ve been banned, using Google as an example, you can visit their Webmaster Tools section to check for any crawling errors. It is wise to check your site thoroughly against the Webmaster Guidelines to ensure you have adhered properly to Google’s rules before submitting a re-inclusion request via the Webmaster Tools area. If the problem is simple or the fault of the spiders, you will be re-indexed very quickly. However, if you have unwittingly slipped into any optimization activity that is disapproved by Google, it could take months to get you back into the SERPs. It should be said, that it is highly unlikely for someone who is not deliberately trying to fool the search bots to get banned. Real spammers know the risks and expect to be banned within a fairly short space of time, but for everyone else, it is a good idea to refresh your knowledge of the search engine guidelines on a regular basis.

Strategic keyword placement increases site visibility

1. Create order: On-page optimization includes making the structure of your site clean, easy to navigate and highly visible (through correct code, links and meta data) to the search engine spiders. Your keywords should support these efforts. Stuffing all your keywords into your home page is not effective. Think about how the search engines crave theme and order; and how a web user refines their search the closer they get to purchase (see our Search Process article). Your website should reflect this hierarchy of cascading information. Your home page, by its very nature, is likely to be fairly generic, and more appropriate for using your broader theme or category keyword descriptions. The rest of your site should be structured to explore more specific sub-categories within a clear framework. If you have a travel site for example, you may be talking about holidays in general on the home page, subsequent pages might talk about certain types of holiday i.e. beach, with sub-pages investigating products or locations. You can then create keyword clusters throughout your site that will generate pockets of strength and relevance for the search engines, resulting in more of your site being crawled more often. Be careful not to dilute your keywords by trying to target too many unrelated words in one area.

Strategically clustering your keywords hierarchically throughout your site will also help with inbound links. Bloggers and webmasters prefer to link to particular articles or pages rather than generic home pages and deep links to certain sections of your site will lend further relevance and strength overall.

2. Get to the point: When it comes to SEO, it pays to be direct. Highly creative writing on a website is all well and good, but a search engine does not currently have the ability to make conceptual or suggestive leaps like that of a human brain. Search engines do not read between the lines. For example, if you are talking aspirationally about your holiday company using phrases such as ‘time to unwind’ etc, the chances are the search engines will not relate you to holidays, but you might get returned in a search for easy listening music. To that end, your headlines and priority text should contain your targeted words and phrases. The words that are important to you and directly relevant to your site should be your focus as they will tell the search engines what your article or page is about. Using clear explanation and words related to your theme will help them do this quicker. Examples of proper keyword placement include:

Keywords in URLs: Your URL is a key descriptor of your site. If possible you should have your keyword forming part of your URL in order to help with SERPs rankings.

Keywords in title tags: The search engines place more importance on page titles than subsequent text. Putting your keyword at the start of your title will immediately tell the search engine’s what that page is about. What’s written in your title tag also shows up as the page title in the search results listing.

Keywords in headings: Headlines also carry more weight in the search engine’s eyes. Using HTML H1, H2 etc headings makes your content more visible to the spiders and enables clearer categorisation of your site content.

Keywords in anchor text: Using your keywords in link descriptions, i.e. your anchor text is another way of getting your keywords into your site content without appearing repetitive. Anchor texts provide more clues to the search engines as to the content of your linking pages.

Keywords in body text: While we’ve established you should be clear and ‘themed’ as far as the search engines go, you should be careful in terms of overdoing it with overuse of your keywords. Not only will this provide difficult reading for the user, but the search engines will punish excessive ‘keyword stuffing’. Keyword ‘density’ is a term used to describe the percentage of keywords you have in an article in proportion to other text. Over 10% of core keywords in an article is thought to start looking suspicious to the search engines. But common sense needs to feature here anyway, if a piece becomes difficult or frustrating to read, there is less likelihood of it ever getting linked to so you really need to be writing for the web user and not the search engines.

Keywords in alt tags: The search engines cannot read images. What may look nice or be highly explanatory to a web user, may be completely invisible to the spiders. It is important to include an image description with any image you post to make it ‘readable’ for the search engines, and adding your keyword to this makes sense. However, you need avoid the temptation to stuff the alt tag with keywords and should make sure the description is still relevant to the image itself.

Keywords in meta tags: This is less important, especially as far as Google is concerned. However, writing a good meta description for each page is good practice and lends some added relevancy to each page. Again, beware of keyword stuffing as it will do more harm than good.

Read more about on-page optimization and how to make your site visible and accessible to the search engines.

Link baiting helps drive traffic

Creating reasons to visit
There are a number of ways in which you can seduce other bloggers and webmasters to link to you.

1. Staying current: Where are you getting your information from? If you are compiling articles by sweeping the web for information, that’s fine as long as you put your own spin on it and don’t directly duplicate content. However, it won’t necessarily get you noticed.
In order to stay fresh and be able to create a piece that will resonate through the web, it is a good idea to spend some time in the online community; searching latest blogs participating in forums relevant to your field and so forth. By properly listening to what people are saying in these forums i.e. what they’d like to see more of, pet hates and current issues etc, you can get some good fodder for your next post. Being active in the social networks i.e. Digg can kick-start your popularity and lend you some visibility to start with.

2. Capitalizing on partnerships: Think about whether you could team up with an expert in your field. This will lend immediate credibility to you and your site. A joint venture or even securing a quote from said expert can work. You could do a feature on them and link to their site. If the article is good, you could get the expert linking back to you for self-promotion.

3. Being funny or unique: Comedy is arguably the primary reason for a web feature to go viral. For example, some of the most highly visited youtube videos have consisted of nothing more than a baby Panda’s sneezing (to its mum’s great surprise). Nothing groundbreaking there, but it did manage to tickle the global nation for a while.

If you can’t think of something funny, or it seems inappropriate to do so, it also pays to be unique. As mentioned above, monitoring blog subjects will help give you ideas, but you could have a personal view or opinion that might spa rk debate. Something that has neither been proved or disproved in the past stands a good chance of getting talked about in the present.

4. Link baiting: While all content designed to get links could be referred to as link baiting, it is more common that you associate the phrase with giving away free stuff. Providing one-stop solutions e.g. complete how to guides/top 10 lists etc, free tools such as BMI calculators, search tools and so on, is a great way to attract links. A website can make itself more useful by providing access to such tools to web users, but even more effective is if you can provide a tool that a blogger or webmaster would be interested in. This will provide an excellent source of links.

5. Avoiding overt commerciality: Websites that feature lots of ads or overtly sell within their content are generally avoided by bloggers or webmasters. A piece that appears purely informative is far more likely to attract genuine links of quality.

While this article only picks up on a few of the techniques that can be employed to generate interesting content, the overall message should be that generating less high quality pages is far more valuable to the virtual community, and your site, than having hundreds of irrelevant filler pages. The goal with online content is to become an authority or reference point for a subject in order to get linked to time and again.

Google AdWords Keyword Tool
This remains a highly popular tool for generating keyword ideas related to your product, service or topic.

Wordtracker Free Keywords
This does the same thing as AdWords but with less option to refine your search.

Keyword Research Tool
Found on Seobook.com and powered by Wordtracker, this tool provides search figures across the major search engines, Google, Yahoo and MSN and includes click-throughs to several keyword suggestion tools.

Quintura
The Quintura tool will give you an insight into keyword maps based on popular search terms. Can be useful in brainstorming keyword ideas.

Blogpulse
Blogpulse lets you track trends on keywords, phrases and links by monitoring the buzz surrounding these search terms. Measured in percentage of blogs posted on those subjects within your specified time frame.

Longtail Keywords Tool
Find out what tail terms are suggested for your site or ads using the Longtail Keywords Tool. Can be a bit picky and does not always yield results but can give you an idea of what your competitors might target even if your site doesn’t show up.

Digital Point’s Keyword Tool
Gives keyword suggestions for your topic with options to overlay specific regions for localised results.

This Q&A attempts to answer some of the most commonly asked questions regarding linkbuilding in SEO.

What is a natural link structure?
How do I get inbound links?
What’s a reciprocal link?
Why bother with outbound links?
What is deep-linking?

Black hat SEO is much more underhand. Content is created deliberately for the search engines in order to trick the spiders into thinking that the content they are searching is relevant to the request, thus gaining an inappropriately high rank. What the search engine bots see is not what the web user views. Black hats generally know their site will be penalised or banned when they’re finally discovered and aim to reap top positions in a short space of time, usually in order to make quick money. Black hat SEO is also known as spamdexing, generally referring to all spam techniques that are disapproved of by the search engines.

Avoid the temptations of quick SEO wins

Bait and switch
Cloaking
Doorway pages
Hidden links
Domain name stuffing
Cross-linking
Auto-generating content
Spamming forums and blogs
Hijacking sites
Link farms
Duplicate content

1. Keyword Analysis: The first stage involves analysis of your product or service and identification of the search terms, or keywords, that could most effectively deliver the largest volume and highest quality traffic to your site, dependent on your phase of development.

2. On-page optimization: Once your keywords are determined and you have a development plan in place, stage two involves the on-page optimization of your site for the keywords you are targeting, as well as ensuring it is accessible to the search engine spiders. This includes optimizing content, headlines, URLs, code, meta data and so forth.

3. Linkbuilding: Once you are happy with your site offering and structure, you can move onto the third, and arguably most crucial, stage of development which is building links. The best site in the world will not be visible to the search engines if there are no other sites pointing to it. Securing a mix of inbound, outbound and reciprocal links is a key factor in establishing a ranking in the search engine results.

4. Reporting: And last but not least, monitoring your SEO activity will enable you to spot trends and make necessary or cost-efficient adjustments to your site and keywords in response to the fluid, ever-changing nature of the web.

It is possible to summarise the four stages of the SEO process in a simplified visual representation.

4 Stage SEO Process

1. Keyword Analysis
-Identify search potential (traffic) of keywords relative to product
-Assess competition & ROI projections on keywords
-Recommend keyword targeting

2. On-page Analysis/Optimization
-Assess accessibility of site to search engines including code and meta data
-Optimize content relative to keywords
-Optimize internal link structure

3. Linkbuilding
-Develop linkbuilding strategy to add strength to your site
-Inbound links via content/requests
-Directories
-Link exchanges/buying

4. Reporting
-Monitor traffic generated by keyword
-Track rank in search engine results
-Review user behaviour, conversions, ROI

Keyword research helps spread the risk

Know your product

Assess your business to determine whether your products/services are widely known about. If you are operating in a niche, or emerging market, some level of education may be required. If people don’t know the terms of your business in order to search for you, some keyword manipulation has to be done to ensure you can make yourself visible. This may involve targeting closely related or themed keywords – the key will be keeping content relevant for the search engines and web users.

If your products or services can be localised i.e. an additional geographical search criteria is added to the search term in order to show results within the region of the user, it makes sense to add locations to your keywords. Similarly, if your products fluctuate in popularity according to season, the addition of seasonal descriptors is going to be advisable at these times.

Look at the competition

Spend some time assessing the competition. Enter a few search terms and have a nose around your competitor’s sites. Checking the keywords in urls and primary content will give you an idea of what they are optimising for and help you appreciate the level and angle of the competition. Apply learnings from trend analysis and keyword tools to add a clearer picture of what web users are most commonly searching for. Consider this in terms of your offering – are those competing in your arena big brands or small companies, are their websites large, polished and well established and how do you compare? A certain degree of honesty is required in adequately assessing your business. You need to determine whether you have the power at this particular point in time to target keywords that may be monopolised by stronger companies. Knowing your limitations will help you select more realistic keywords.

Assess user error

It sounds very simple but when you are considering your keywords you should remember to consider popular misspellings of words. Keyword tools can help you assess this by showing the volume of searches performed on similar or misspelt words close to your own. It is good practice to optimize for these words as well as your key term.

Measure your success

Importantly, once you have assessed and selected the most appropriate keywords for your site and optimised for them, you should still monitor the above factors. Competition is constantly changing and fads, latest news and seasons will always affect searches. As your site grows and the market changes you will need to tinker with your keywords and keyphrases (known as keyword stemming) and even change them to ensure you stay relevant and effective within your marketplace as time goes on.
Search for free keyword research tools.

Find out more about effectively optimizing your page using your selected keywords in our On-page section.