| Presentation | 2021-03-04 Estimation of imagined speech from electrocorticogram with an encoder-decoder model Kotaro Hayashi, Shuji Komeiji, Takumi Mitsuhashi, Yasushi Iimura, Hiroharu Suzuki, Hidenori Sugano, Koichi Shinoda, Toshihisa Tanaka, |
|---|---|
| PDF Download Page |  PDF download Page Link |
| Abstract(in Japanese) | (See Japanese page) |
| Abstract(in English) | Recent advances in signal processing and machine learning technologies have made it possible to estimate and reconstruct speech or text during speaking and listening from invasive electrocorticogram (ECoG). Meanwhile, the estimation of imagined speech has not been successful due to the difficulty in synchronizing the ECoG with the target label. In this paper, we hypothesize that if imagined speech and ECoG are adequately synchronized, speech can be decoded from ECoG as in the case of ECoG during speaking and listening. We designed an experiment in which participants were presented with a screen on which short sentences were projected, and by highlighting the colors of the letters one by one, the timing and speed of imagination could be controlled. The ECoG was recorded during the three tasks of imagining speech, listening to speech, and speaking. Moreover, we recorded the speech presented in the listening task and the participants' speech in the speaking task. From the measured ECoG, we built an encoder-decoder model using the Mel cepstrum coefficients of the speech during the speaking and listening tasks to infer the imagined speech. The best character error rate of about 17% was achieved for sentences decoded from the imagined ECoG. |
| Keyword(in Japanese) | (See Japanese page) |
| Keyword(in English) | Electrocorticogram (ECoG) / Imaginary speech / Brain-computer interface (BCI) / Encoder-decoder model |
| Paper # | EA2020-87,SIP2020-118,SP2020-52 |
| Date of Issue | 2021-02-24 (EA, SIP, SP) |
| Conference Information | |
| Committee | EA / US / SP / SIP / IPSJ-SLP |
|---|---|
| Conference Date | 2021/3/3(2days) |
| Place (in Japanese) | (See Japanese page) |
| Place (in English) | Online |
| Topics (in Japanese) | (See Japanese page) |
| Topics (in English) | Speech, Engineering/Electro Acoustics, Signal Processing, Ultrasonics, and Related Topics |
| Chair | Kenichi Furuya(Oita Univ.) / Hikaru Miura(Nihon Univ.) / Hisashi Kawai(NICT) / Kazunori Hayashi(Kyoto Univ.) / 北岡 教英(豊橋技科大) |
| Vice Chair | Yoshinobu Kajikawa(Kansai Univ.) / Kentaro Matsui(NHK) / Jun Kondo(Shizuoka Univ.) / Yoshikazu Koike(Shibaura Inst. of Tech.) / / Yukihiro Bandou(NTT) / Toshihisa Tanaka(Tokyo Univ. Agri.&Tech.) |
| Secretary | Yoshinobu Kajikawa(Univ. of Tokyo) / Kentaro Matsui(NTT) / Jun Kondo(Doshisha Univ.) / Yoshikazu Koike(Tohoku Univ.) / (Univ. of Tokyo) / Yukihiro Bandou(Waseda Univ.) / Toshihisa Tanaka(Hosei Univ.) / (Waseda Univ.) |
| Assistant | Yukou Wakabayashi(Tokyo Metropolitan Univ.) / Tatsuya Komatsu(LINE) / Shinnosuke Hirata(Tokyo Inst. of Tech.) / Yusuke Ijima(NTT) / Yuichi Tanaka(Tokyo Univ. Agri.&Tech.) |
| Paper Information | |
| Registration To | Technical Committee on Engineering Acoustics / Technical Committee on Ultrasonics / Technical Committee on Speech / Technical Committee on Signal Processing / Special Interest Group on Spoken Language Processing |
|---|---|
| Language | JPN |
| Title (in Japanese) | (See Japanese page) |
| Sub Title (in Japanese) | (See Japanese page) |
| Title (in English) | Estimation of imagined speech from electrocorticogram with an encoder-decoder model |
| Sub Title (in English) | |
| Keyword(1) | Electrocorticogram (ECoG) |
| Keyword(2) | Imaginary speech |
| Keyword(3) | Brain-computer interface (BCI) |
| Keyword(4) | Encoder-decoder model |
| 1st Author's Name | Kotaro Hayashi |
| 1st Author's Affiliation | Tokyo University of Agriculture and Technology(TUAT) |
| 2nd Author's Name | Shuji Komeiji |
| 2nd Author's Affiliation | Tokyo University of Agriculture and Technology(TUAT) |
| 3rd Author's Name | Takumi Mitsuhashi |
| 3rd Author's Affiliation | Juntendo University(Juntendo Univ.) |
| 4th Author's Name | Yasushi Iimura |
| 4th Author's Affiliation | Juntendo University(Juntendo Univ.) |
| 5th Author's Name | Hiroharu Suzuki |
| 5th Author's Affiliation | Juntendo University(Juntendo Univ.) |
| 6th Author's Name | Hidenori Sugano |
| 6th Author's Affiliation | Juntendo University(Juntendo Univ.) |
| 7th Author's Name | Koichi Shinoda |
| 7th Author's Affiliation | Tokyo Institute of Technology(TokyoTech) |
| 8th Author's Name | Toshihisa Tanaka |
| 8th Author's Affiliation | Tokyo University of Agriculture and Technology(TUAT) |
| Date | 2021-03-04 |
| Paper # | EA2020-87,SIP2020-118,SP2020-52 |
| Volume (vol) | vol.120 |
| Number (no) | EA-397,SIP-398,SP-399 |
| Page | pp.pp.164-169(EA), pp.164-169(SIP), pp.164-169(SP), |
| #Pages | 6 |
| Date of Issue | 2021-02-24 (EA, SIP, SP) |