There are two words that I would like to cover today. They were prompted by an article written by SEO-speedwagon. The two terms are Index and Crawling.

Crawling: “is the process of an engine requesting — and successfully downloading — a unique URL.”

What this means: Google, Yahoo and other search engines have the ability to surf the internet. To surf the internet they use a small program called a spider or robot. What these spiders or robots do is read different pages that they come across. They follow links to find the page that they read and that is one reason why links are so important. When they arrive at a page the spider starts at the top and reads all the content on that page. If a spider follows a link to a specific page and receives an error then the spider cannot crawl that particular page.

Index: “is the result of successful crawling.”

What this means: Once a spider or robots has read everything on a particular page it adds all of that information to what is known as its index. A search engine index is very similar to that used in a book. It is a quick reference guide as to where the information is and what it may be about. Once a page has been successfully indexed and made available when a search is performed the page will appear in the Search Engine Results Pages.

Problems that may be encountered when a spider is crawling pages are, it may be blocked by a robots.txt file or the server could be switched off or coding on the page may stop the search engines from reading the page correctly. There is normally a delay in time between a successful crawl and the page being displayed in the results of a search engine. This delay depends on the site and can be as little as a few minutes, for some sites that the spiders trust and access regularly for example a news site. For a new site without any page rank it could be anything up to a week before the page is available in the search results of Google and other search engines.

There are a number of tools that can help in identifying what sites are getting the majority of traffic for a search term. I have taken the search term “Hobart” and looked at the competition through Hitwise. Below is the chart of who is getting the most traffic from this search local term.

This phrase seems quite backward as we rarely provide an answer before some one has asked a question.

Writing an article however tries to answer the question before the reader has asked it. When writing anything you are preempting what the reader is going to ask and then you try and phrase your information around the question, even if this process is subconscious.

The internet has become a place for adding articles and has given many people a place where they can write, share information and answer questions before they are asked.

If you would like people to read your writing, you at least need to identify what the question you are going to address is. If your article rambles and has no relevance to a readers question they will stop reading.

On the internet it is possible to do question research. It is easy to do this research if you consider the keywords people are typing in on the various search engines as the questions they are asking.

If we classify internet users into two categories,

1. browsers
2. hunters

it is easier to identify what type of questions the internet searcher is going to be asking.

An internet user that is browsing is searching for something, however what they are looking for is not necessarily understood or quantifiable if you asked them. The hunter on the other hand knows exactly what they are looking for and will use specific words that they believe best represents what they are trying to find.

For example if I was looking for “Indian chicken recipe” you can be pretty sure that what I am after is a recipe that contains chicken and is of an Indian flavor or curry. However if I typed in “recipe” I could be looking for almost any type of recipe, it may not even been food related it could be a “recipe for a disaster.

This identifying key phrases for an internet article is very important. It allows the writer to address the topic correctly. It also allows the search engines to categories the article correctly, which is vital if you want your article to be found for a question that an internet searcher is asking.

When someone types in a long phrase like “Indian chicken recipe” this is known as long tail. A short phrase like “recipe” is known as a short tail phrase. When considering all the phrases typed into the search engines there are many more short tail phrases then long tail phrases.

If you are targeting a long tail phrase you will probably get less hits but the people that come looking at the article or content that you have written and posted will be qualified or more interested in what you have to say. When targeting short tail phrases your spread of area is much larger and the proportion of users who won’t be interested in what you have to say will probably be much greater.

So when writing content for an internet site or an article, define the question, define the scope. Are you going to get specific or targeted? Once you have decided these things the article will be much easier to write and easier for the search engines to classify. And then people will listen to what you have to say.

Goals should be broken down into very achievable objectives. One of the best places to start in identifying what goals for a website ought to be is to start with the keywords. If done correctly the keywords will give specific targets to aim at. The next area is to identify where the traffic is to come from. This can be search engines, directories, web 2.0 sites or other authoritative sites that already have traffic.

Goals should always be written down. An excel spreadsheet is a good place to start. A good first step is to list out 5 key phrases that the site is aiming at being found for. In the next column list out where the traffic is to come from for each key phrase.

In the third column list the activities that need to be performed to achieve that goal. These activities may include submitting to directories using certain key phrases. Getting links using the key phrases identified. Writing articles with these key phrases. Changing the copy to incorporate the keywords listed under each goal. Using other mediums such as social search, RSS, audio or video.

As each of these goals are achieved review how long it took to achieve it. Note any difficulties and things that may have helped in achieving the goal. And then most importantly set new goals to work to.