| Presentation | 2023-01-22 Analysis of Role of Multimodal Information and Physiological Signals in Self-Reported and Third-Party Sentiment Estimation Shun Katada, Shogo Okada, Kazunori Komatani, |
|---|---|
| PDF Download Page |  PDF download Page Link |
| Abstract(in Japanese) | (See Japanese page) |
| Abstract(in English) | A dialogue system needs to adapt its utterance by estimating a user's internal state. Most of the previous multimodal sentiment analyses use third-party sentiment label; however, a user's self-sentiment annotated by the user themselves is often ignored. Physiological signals can detect subtle changes in self-sentiment. While the relationship between text, acoustic and visual modalities has been analyzed, the relationship between physiological signals and the three modalities has not been well discussed. In this study, we construct sentiment estimation models based on text, acoustic, visual, and physiological information, and compared them for the two types of sentiment estimation. As a result, the fusion of the text and physiological modalities is effective for self-sentiment estimation. In contrast, the fusion of the text and audiovisual modalities is effective for third-party sentiment estimation. Thus, it is suggested that effective modalities for sentiment estimation depend on the type of sentiment labels. |
| Keyword(in Japanese) | (See Japanese page) |
| Keyword(in English) | self-sentiment / third-party sentiment / multimodal information / physiological signal |
| Paper # | HCS2022-70 |
| Date of Issue | 2023-01-14 (HCS) |
| Conference Information | |
| Committee | HCS |
|---|---|
| Conference Date | 2023/1/21(2days) |
| Place (in Japanese) | (See Japanese page) |
| Place (in English) | Kyoto Institute of Technology |
| Topics (in Japanese) | (See Japanese page) |
| Topics (in English) | Physical Communication, etc. |
| Chair | Tomoko Kanda(Osaka Inst. of Tech.) |
| Vice Chair | Yugo Hayashi(Ritsumeikan Univ.) / Masashi Komori(Osaka Electro-Comm. Univ.) |
| Secretary | Yugo Hayashi(Nihon Univ.) / Masashi Komori(Kanagawa Univ.) |
| Assistant | HUANG HUNGHSUAN(Univ. of Fukuchiyama) / Jun Ichikawa(Shizuoka Univ.) / Kazuki Takashima(Tohoku Univ.) / Hiroto Saito(Tokyo Denki Univ.) / Ryo Ishii(NTT) / Yoshimasa Ohmoto(Shizuoka Univ.) |
| Paper Information | |
| Registration To | Technical Committee on Human Communication Science |
|---|---|
| Language | JPN |
| Title (in Japanese) | (See Japanese page) |
| Sub Title (in Japanese) | (See Japanese page) |
| Title (in English) | Analysis of Role of Multimodal Information and Physiological Signals in Self-Reported and Third-Party Sentiment Estimation |
| Sub Title (in English) | |
| Keyword(1) | self-sentiment |
| Keyword(2) | third-party sentiment |
| Keyword(3) | multimodal information |
| Keyword(4) | physiological signal |
| 1st Author's Name | Shun Katada |
| 1st Author's Affiliation | Japan Advanced Institute of Science and Technology(JAIST) |
| 2nd Author's Name | Shogo Okada |
| 2nd Author's Affiliation | Japan Advanced Institute of Science and Technology(JAIST) |
| 3rd Author's Name | Kazunori Komatani |
| 3rd Author's Affiliation | Osaka University(Osaka Univ.) |
| Date | 2023-01-22 |
| Paper # | HCS2022-70 |
| Volume (vol) | vol.122 |
| Number (no) | HCS-349 |
| Page | pp.pp.86-91(HCS), |
| #Pages | 6 |
| Date of Issue | 2023-01-14 (HCS) |