| 講演名 | 2013/7/15 How Intuitive Are Diversified Search Metrics? Concordance Test Results for the Diversity U-measures , |
|---|---|
| PDFダウンロードページ | PDFダウンロードページへ |
| 抄録(和) | |
| 抄録(英) | For the past few decades, ranked retrieval (e.g. web search) has been evaluated using rank-based evaluation metrics such as Average Precision and normalised Discounted Cumulative Gain (nDCG). These metrics discount the value of each retrieved relevant document based on its rank. The situation is similar with diversified search which has gained popularity recently: diversity metrics such as a-nDCG, Intent-Aware Expected Reciprocal Rank (ERR-IA) and D#-nDCG are also rank-based. These widely-used evaluation metrics just regard the system output as a list of document IDs, and ignore all other features such as snippets and document full texts of various lengths. The recently-proposed U-measure framework of Sakai and Dou uses the amount of text read by the user as the foundation for discounting the value of relevant information, and can take into account the user's snippet reading and full text reading behaviours. The present study compares the diversity versions of U-measure (D-U and U-IA) with state-of-the-art diversity metrics in terms of how "intuitive" they are: given a pair of ranked lists, we quantify the ability of each metric to favour the more diversified and more relevant list by means of the concordance test. Our results show that while D#-nDCG is the overall winner in terms of simultaneous concordance with diversity and relevance, D-U and U-IA statistically signif-icantly outperform other state-of-the-art metrics. Moreover, in terms of concordance with relevance alone, D-U and U-IA significantly outperform all rank-based diversity metrics. These results suggest that D-U and U-IA are not only more realistic than rank-based metrics but also intuitive, i.e., that they measure what we want to measure. |
| キーワード(和) | |
| キーワード(英) | diversity / evaluation / intents / TREC / subtopics / web search |
| 資料番号 | Vol.2013-DBS-157 No.12,Vol.2013-IFAT-111 No.12 |
| 発行日 |
| 研究会情報 | |
| 研究会 | DE |
|---|---|
| 開催期間 | 2013/7/15(から1日開催) |
| 開催地（和） | |
| 開催地（英） | |
| テーマ（和） | |
| テーマ（英） | |
| 委員長氏名（和） | |
| 委員長氏名（英） | |
| 副委員長氏名（和） | |
| 副委員長氏名（英） | |
| 幹事氏名（和） | |
| 幹事氏名（英） | |
| 幹事補佐氏名（和） | |
| 幹事補佐氏名（英） | |
| 講演論文情報詳細 | |
| 申込み研究会 | Data Engineering (DE) |
|---|---|
| 本文の言語 | ENG |
| タイトル（和） | |
| サブタイトル（和） | |
| タイトル（英） | How Intuitive Are Diversified Search Metrics? Concordance Test Results for the Diversity U-measures |
| サブタイトル（和） | |
| キーワード(1)（和/英） | / diversity |
| 第 1 著者 氏名（和/英） | / TETSUYA SAKAI |
| 第 1 著者 所属（和/英） | Microsoft Research Asia China |
| 発表年月日 | 2013/7/15 |
| 資料番号 | Vol.2013-DBS-157 No.12,Vol.2013-IFAT-111 No.12 |
| 巻番号（vol） | vol.113 |
| 号番号（no） | 150 |
| ページ範囲 | pp.- |
| ページ数 | 6 |
| 発行日 | |