Presentation 2008-06-20
Gathering Over 10 Billion of Web Pages and its Applications
Hayato YAMANA,
PDF Download Page PDF download Page Link
Abstract(in Japanese) (See Japanese page)
Abstract(in English) The number of Web pages distributed from Web servers is estimated about 53.7 billion as of Oct. 2005. We had gathered 14,456,201,906 Web pages from 5,548 Web servers during Jan. 2004 to July 2006. It had been conducted as part of e-Society project which is one of MEXT, Ministry of Education, Culture, Sports, Science and Technology, leading projects. Speedup of crawling Web pages conflicts with Web-site friendly crawling, however, both are indispensable for gathering Web pages. In the project, we have studied and proposed a dynamic delay adjustment scheme for accessing Web servers to prevent hammering them. Moreover, many know-how including access control for Web servers have been studied. As for the analysis, written language distribution, top level domain distribution and geographical location distribution of 10.7 billion of Web pages were calculated. Besides that, back-link analysis and PageRank calculation among Web servers were done. In this talk, I will introduce a crawling technique for gathering large amount of Web pages and applications for best use of gathered Web pages.
Keyword(in Japanese) (See Japanese page)
Keyword(in English) Web repository / Web crawler / Web analysis / Link analysis
Paper # DE2008-17,PRMU2008-35
Date of Issue

Conference Information
Committee DE
Conference Date 2008/6/12(1days)
Place (in Japanese) (See Japanese page)
Place (in English)
Topics (in Japanese) (See Japanese page)
Topics (in English)
Chair
Vice Chair
Secretary
Assistant

Paper Information
Registration To Data Engineering (DE)
Language JPN
Title (in Japanese) (See Japanese page)
Sub Title (in Japanese) (See Japanese page)
Title (in English) Gathering Over 10 Billion of Web Pages and its Applications
Sub Title (in English)
Keyword(1) Web repository
Keyword(2) Web crawler
Keyword(3) Web analysis
Keyword(4) Link analysis
1st Author's Name Hayato YAMANA
1st Author's Affiliation Science and Engineering, Waseda University()
Date 2008-06-20
Paper # DE2008-17,PRMU2008-35
Volume (vol) vol.108
Number (no) 93
Page pp.pp.-
#Pages 87
Date of Issue