2009 IncorporatingSiteLevelKnowledge

Subject Headings:

Notes

Categories and Subject Descriptors: H.3.3 Information Storage and Retrieval: Information Search and Retrieval - clustering, Information Filtering.
General Terms: Algorithms, Performance, Experimentation

We study in this paper the problem of incremental crawling of web forums, which is a very fundamental yet challenging step in many web applications. Traditional approaches mainly focus on scheduling the revisiting strategy of each individual page. However, simply assigning different weights for different individual pages is usually inefficient in crawling forum sites because of the different characteristics between forum sites and general websites. Instead of treating each individual page independently, we propose a list-wise strategy by taking into account the site-level knowledge. Such site-level knowledge is mined through reconstructing the linking structure, called sitemap, for a given forum site. With the sitemap, posts from the same thread but distributed on various pages can be concatenated according to their timestamps. After that, for each thread, we employ a regression model to predict the time when the next post arrives. Based on this model, we develop an efficient crawler which is 260% faster than some state-of-the-art methods in terms of fetching new generated content; and meanwhile our crawler also ensure a high coverage ratio. Experimental results show promising performance of Coverage, Bandwidth utilization, and Timeliness of our crawler on 18 various forums.

,

	Author	volume	Date Value	title	type	journal	titleUrl	doi	note	year
2009 IncorporatingSiteLevelKnowledge	Jiang-Ming Yang Rui Cai Chunsong Wang Hua Huang Lei Zhang Wei-Ying Ma			Incorporating Site-level Knowledge for Incremental Crawling of Web Forums: A List-wise Strategy		KDD-2009 Proceedings		10.1145/1557019.1557166		2009