URL Canonicalization Task

See: Canonicalization Task, URL, URL Normalization.

References

http://en.wikipedia.org/wiki/URL_canonicalization
- URL normalization (or URL canonicalization) is the process by which URLs are modified and standardized in a consistent manner. The goal of the normalization process is to transform a URL into a normalized or canonical URL so it is possible to determine if two syntactically different URLs are equivalent.
- Search engines employ URL normalization in order to assign importance to web pages and to reduce indexing of duplicate pages. Web crawlers perform URL normalization in order to avoid crawling the same resource more than once. Web browsers may perform normalization to determine if a link has been visited or to determine if a page has been cached.
(Wikipedia, 2009) ⇒ http://en.wikipedia.org/wiki/Canonicalization
- … In web search and search engine optimization (SEO), URL canonicalization deals with web content that has more than one possible URL. Having multiple URLs for the same web content can cause problems for search engines - specifically in determining which URL should be shown in search results. ^[1]
- Example:


http://www.wikipedia.com
http://wikipedia.com
http://www.wikipedia.com/
http://www.wikipedia.com/?source=asdf

- All of these URLs point to the homepage of Wikipedia, but a search engine will only consider one of them to be the canonical form of the URL.