| Presentation | 2022-07-04 Unsupervised Learning of a Dynamic Task Ordering Model for Crowdsourcing Ryo Yanagisawa, Susumu Saito, Teppei Nakano, Tetsunori Kobayashi, Tetsuji Ogawa, |
|---|---|
| PDF Download Page |  PDF download Page Link |
| Abstract(in Japanese) | (See Japanese page) |
| Abstract(in English) | An unsupervised learning method for a dynamic task ordering model that optimizes the number of orders according to the difficulty of the data was proposed as a framework for efficiently ensuring annotation quality through crowdsourcing. Since responses collected by crowdsourcing contain errors, the responses were collected from multiple workers for each sample and then aggregated by majority voting to ensure reliability. However, since the monetary cost increases as the number of orders increases, it is desirable to reduce the number of workers who perform majority voting while maintaining the high accuracy of the final label. Therefore, we focus on a dynamic task ordering model that continues to place orders to workers until the variation in responses becomes sufficiently small, based on the assumption that the smaller the variation in responses by multiple workers, the more reliable the majority decision is. The present study proposed a method for unsupervised learning of model parameters such that label errors and ordering costs are minimized. Experimental comparisons on an annotation task for livestock surveillance images demonstrated the effectiveness of the proposed method: it achieved performance comparable to supervised learning and significantly reduced the number of orders without significantly degrading accuracy compared to simple majority voting, which emphasizes accuracy. |
| Keyword(in Japanese) | (See Japanese page) |
| Keyword(in English) | Crowdsourcing / Quality control / Unsupervised learning / Dynamic task ordering |
| Paper # | AI2022-14 |
| Date of Issue | 2022-06-27 (AI) |
| Conference Information | |
| Committee | AI |
|---|---|
| Conference Date | 2022/7/4(1days) |
| Place (in Japanese) | (See Japanese page) |
| Place (in English) | |
| Topics (in Japanese) | (See Japanese page) |
| Topics (in English) | |
| Chair | Yuichi Sei(Univ. of Electro-Comm.) |
| Vice Chair | Yuko Sakurai(AIST) / Tadachika Ozono(Nagoya Inst. of Tech.) |
| Secretary | Yuko Sakurai(Tokyo Univ. of Agriculture and Technology) / Tadachika Ozono(Toho Univ.) |
| Assistant | Kazutaka Matsuzaki(Chuo Univ.) |
| Paper Information | |
| Registration To | Technical Committee on Artificial Intelligence and Knowledge-Based Processing |
|---|---|
| Language | JPN |
| Title (in Japanese) | (See Japanese page) |
| Sub Title (in Japanese) | (See Japanese page) |
| Title (in English) | Unsupervised Learning of a Dynamic Task Ordering Model for Crowdsourcing |
| Sub Title (in English) | |
| Keyword(1) | Crowdsourcing |
| Keyword(2) | Quality control |
| Keyword(3) | Unsupervised learning |
| Keyword(4) | Dynamic task ordering |
| 1st Author's Name | Ryo Yanagisawa |
| 1st Author's Affiliation | Waseda University(Waseda Univ.) |
| 2nd Author's Name | Susumu Saito |
| 2nd Author's Affiliation | Intelligent Framework Lab Inc.(ifLab Inc.) |
| 3rd Author's Name | Teppei Nakano |
| 3rd Author's Affiliation | Intelligent Framework Lab Inc.(ifLab Inc.) |
| 4th Author's Name | Tetsunori Kobayashi |
| 4th Author's Affiliation | Waseda University(Waseda Univ.) |
| 5th Author's Name | Tetsuji Ogawa |
| 5th Author's Affiliation | Waseda University(Waseda Univ.) |
| Date | 2022-07-04 |
| Paper # | AI2022-14 |
| Volume (vol) | vol.122 |
| Number (no) | AI-94 |
| Page | pp.pp.72-76(AI), |
| #Pages | 5 |
| Date of Issue | 2022-06-27 (AI) |