WDC Hyperlink Dataset
Jump to navigation
Jump to search
A WDC Hyperlink Dataset is a Web hyperlink dataset that is a Web Data Commons dataset.
- AKA: Web Data Commons Hyperlink Dataset.
- Context:
- It can be used to describe a Web Graph Topology.
- Example(s):
- See: WDC Web Tables Dataset.
References
2014
- http://webdatacommons.org/hyperlinkgraph/index.html
- This page provides two large hyperlink graph for public download. The graphs have been extracted from the 2012 and 2014 versions of the Common Crawl web corpera. The 2012 graph covers 3.5 billion web pages and 128 billion hyperlinks between these pages. To the best of our knowledge, the graph is the largest hyperlink graph that is available to the public outside companies such as Google, Yahoo, and Microsoft. The2014 graph covers 1.7 billion web pages connected by 64 billion hyperlinks. Below we provide instructions on how to download the graphs as well as basic statistics about their topology.
We hope that the graphs will be useful for researchers who develop
- search algorithms that rank results based on the hyperlinks between pages.
- SPAM detection methods which identity networks of web pages that are published in order to trick search engines.
- graph analysis algorithms and can use the hyperlink graphs for testing the scalability and performance of their tools.
- Web Science researchers who want to analyze the linking patterns within specific topical domains in order to identify the social mechanisms that govern these domains.
- This page provides two large hyperlink graph for public download. The graphs have been extracted from the 2012 and 2014 versions of the Common Crawl web corpera. The 2012 graph covers 3.5 billion web pages and 128 billion hyperlinks between these pages. To the best of our knowledge, the graph is the largest hyperlink graph that is available to the public outside companies such as Google, Yahoo, and Microsoft. The2014 graph covers 1.7 billion web pages connected by 64 billion hyperlinks. Below we provide instructions on how to download the graphs as well as basic statistics about their topology.
- http://webdatacommons.org/hyperlinkgraph/2014-04/download.html
- This page provides detailed download instruction to obtain the hyperlink graph extracted from the Common Crawl 2014 web corpus. The graph covers 1.7 billion web pages and 64 billion hyperlinks between these pages.
We also provide basic statistics about the hyperlink. Please visit the overview page for more information about the provided file formats.
- This page provides detailed download instruction to obtain the hyperlink graph extracted from the Common Crawl 2014 web corpus. The graph covers 1.7 billion web pages and 64 billion hyperlinks between these pages.