| Presentation | 2008-06-20 Gathering Over 10 Billion of Web Pages and its Applications Hayato YAMANA, |
|---|---|
| PDF Download Page |  PDF download Page Link |
| Abstract(in Japanese) | (See Japanese page) |
| Abstract(in English) | The number of Web pages distributed from Web servers is estimated about 53.7 billion as of Oct. 2005. We had gathered 14,456,201,906 Web pages from 5,548 Web servers during Jan. 2004 to July 2006. It had been conducted as part of e-Society project which is one of MEXT, Ministry of Education, Culture, Sports, Science and Technology, leading projects. Speedup of crawling Web pages conflicts with Web-site friendly crawling, however, both are indispensable for gathering Web pages. In the project, we have studied and proposed a dynamic delay adjustment scheme for accessing Web servers to prevent hammering them. Moreover, many know-how including access control for Web servers have been studied. As for the analysis, written language distribution, top level domain distribution and geographical location distribution of 10.7 billion of Web pages were calculated. Besides that, back-link analysis and PageRank calculation among Web servers were done. In this talk, I will introduce a crawling technique for gathering large amount of Web pages and applications for best use of gathered Web pages. |
| Keyword(in Japanese) | (See Japanese page) |
| Keyword(in English) | Web repository / Web crawler / Web analysis / Link analysis |
| Paper # | DE2008-17,PRMU2008-35 |
| Date of Issue |
| Conference Information | |
| Committee | DE |
|---|---|
| Conference Date | 2008/6/12(1days) |
| Place (in Japanese) | (See Japanese page) |
| Place (in English) | |
| Topics (in Japanese) | (See Japanese page) |
| Topics (in English) | |
| Chair | |
| Vice Chair | |
| Secretary | |
| Assistant | |
| Paper Information | |
| Registration To | Data Engineering (DE) |
|---|---|
| Language | JPN |
| Title (in Japanese) | (See Japanese page) |
| Sub Title (in Japanese) | (See Japanese page) |
| Title (in English) | Gathering Over 10 Billion of Web Pages and its Applications |
| Sub Title (in English) | |
| Keyword(1) | Web repository |
| Keyword(2) | Web crawler |
| Keyword(3) | Web analysis |
| Keyword(4) | Link analysis |
| 1st Author's Name | Hayato YAMANA |
| 1st Author's Affiliation | Science and Engineering, Waseda University() |
| Date | 2008-06-20 |
| Paper # | DE2008-17,PRMU2008-35 |
| Volume (vol) | vol.108 |
| Number (no) | 93 |
| Page | pp.pp.- |
| #Pages | 87 |
| Date of Issue | |