| IEICE Technical Committee Submission System Conference Paper's Information |
Online Proceedings [Sign in] Tech. Rep. Archives |
| Go Top Page | Go Previous | [Japanese] / [English] |
| Paper Abstract and Keywords | ||
| Presentation | 2022-06-18 15:00 [Poster Presentation] Worker Filtering Criteria for Subjective Evaluation of Synthesized Voice Sound Quality Using Crowdsourcing Moe Yaegashi (Waseda Univ.), Susumu Saito, Teppei Nakano (Waseda Univ./ifLab.), Tetsuji Ogawa (Waseda Univ.)  SP2022-24 |
|
| Abstract | (in Japanese) | (See Japanese page) |
| (in English) | We investigate the effect of filtering criteria of crowdworkers on the subjective evaluation results of synthesized voice using crowdsourcing. Currently, crowdsourcing has been used for subjective evaluation of synthesized voice. Although it is desirable to remove workers who do not satisfy the client's requirements, worker filtering criteria have not yet been defined. In this study, we focused on subjective evaluation of sound quality (amount of distortion) and examined filtering criteria. In the filtering test, the comparison task was designed so that attributes other than intonation and sound quality were identical in order to enable evaluation of the ability to distinguish differences in sound quality. In order for the worker to understand the difference in sound quality intuitively, we showed the workers the highly distorted voice several times repeatedly at the beginning of the evaluation. We conducted sound quality evaluation experiments on Amazon Mechanical Turk to investigate the effects of the following filtering criteria on the subjective evaluation results: textit{i)} whether the evaluation was focused on the amount of distortion (Understanding of Intent), textit{ii)} whether the responses were consistent (Response Consistency Rate), textit{iii) }whether the responses were given with confidence (Response Confidence). The results showed that measuring the degree of Understanding of Intentions and Response Confidence is effective in worker selection, and this can be achieved by including a few samples that are useful for Understanding of Intention (in this study, low sound quality voice) in the comparison task. | |
| Keyword | (in Japanese) | (See Japanese page) |
| (in English) | crowdsourcing / speech synthesis / worker filtering / subjective evaluation / / / / | |
| Reference Info. | IEICE Tech. Rep., vol. 122, no. 81, SP2022-24, pp. 104-109, June 2022. | |
| Paper # | SP2022-24 | |
| Date of Issue | 2022-06-10 (SP) | |
| ISSN | Online edition: ISSN 2432-6380 | |
| Copyright and reproduction |
All rights are reserved and no part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording, or any information storage and retrieval system, without permission in writing from the publisher. Notwithstanding, instructors are permitted to photocopy isolated articles for noncommercial classroom use without fee. (License No.: 10GA0019/12GB0052/13GB0056/17GB0034/18GB0034) | |
| Download PDF | SP2022-24 |
|
| Conference Information | |
| Committee | SP IPSJ-MUS IPSJ-SLP |
| Conference Date | 2022-06-17 - 2022-06-18 |
| Place (in Japanese) | (See Japanese page) |
| Place (in English) | Online |
| Topics (in Japanese) | (See Japanese page) |
| Topics (in English) | |
| Paper Information | |
| Registration To | SP |
| Conference Code | 2022-06-SP-MUS-SLP |
| Language | Japanese |
| Title (in Japanese) | (See Japanese page) |
| Sub Title (in Japanese) | (See Japanese page) |
| Title (in English) | Worker Filtering Criteria for Subjective Evaluation of Synthesized Voice Sound Quality Using Crowdsourcing |
| Sub Title (in English) | |
| Keyword(1) | crowdsourcing |
| Keyword(2) | speech synthesis |
| Keyword(3) | worker filtering |
| Keyword(4) | subjective evaluation |
| Keyword(5) | |
| Keyword(6) | |
| Keyword(7) | |
| Keyword(8) | |
| 1st Author's Name | Moe Yaegashi |
| 1st Author's Affiliation | Waseda University (Waseda Univ.) |
| 2nd Author's Name | Susumu Saito |
| 2nd Author's Affiliation | Waseda University/Intelligent Framework Lab Inc. (Waseda Univ./ifLab.) |
| 3rd Author's Name | Teppei Nakano |
| 3rd Author's Affiliation | Waseda University/Intelligent Framework Lab Inc. (Waseda Univ./ifLab.) |
| 4th Author's Name | Tetsuji Ogawa |
| 4th Author's Affiliation | Waseda University (Waseda Univ.) |
| 5th Author's Name | |
| 5th Author's Affiliation | () |
| 6th Author's Name | |
| 6th Author's Affiliation | () |
| 7th Author's Name | |
| 7th Author's Affiliation | () |
| 8th Author's Name | |
| 8th Author's Affiliation | () |
| 9th Author's Name | |
| 9th Author's Affiliation | () |
| 10th Author's Name | |
| 10th Author's Affiliation | () |
| 11th Author's Name | |
| 11th Author's Affiliation | () |
| 12th Author's Name | |
| 12th Author's Affiliation | () |
| 13th Author's Name | |
| 13th Author's Affiliation | () |
| 14th Author's Name | |
| 14th Author's Affiliation | () |
| 15th Author's Name | |
| 15th Author's Affiliation | () |
| 16th Author's Name | |
| 16th Author's Affiliation | () |
| 17th Author's Name | |
| 17th Author's Affiliation | () |
| 18th Author's Name | |
| 18th Author's Affiliation | () |
| Speaker | Author-1 |
| Date Time | 2022-06-18 15:00:00 |
| Presentation Time | 120 minutes |
| Registration for | SP |
| Paper # | SP2022-24 |
| Volume (vol) | vol.122 |
| Number (no) | no.81 |
| Page | pp.104-109 |
| #Pages | 6 |
| Date of Issue | 2022-06-10 (SP) |
[Return to Top Page]