| Presentation | 2021-09-16 Construction of Japanese Essay Data for Evaluating Automatic Essay Scoring System Koichi Takeuchi, Masahiro Taguchi, Yoshihiko Inada, Masaya Iizuka, Tasuhiko Abo, Hitoshi Ueda, |
|---|---|
| PDF Download Page |  PDF download Page Link |
| Abstract(in Japanese) | (See Japanese page) |
| Abstract(in English) | We construct Japanese essay data for evaluating automatic essay scoring system. The essays are written by Japanese native speakers for the wide range prompts and all the essays are graded by annotators. In this paper, we give an overview of the essay data and discuss what factors were considered in manual evaluation to clarify the issues involved in creating scored essay data. In manual evaluation, we set four evaluation criteria: comprehension, logic, validity, and writing ability. With a rubric based on the criteria, two raters assign scores to the essays. With evaluation results of inter-rater agreement for prompts, we discuss the stability of human rating on each evaluation criteria depending on the characteristics of prompts. |
| Keyword(in Japanese) | (See Japanese page) |
| Keyword(in English) | automatic essay scoring / Japanese native speakers / evaluation criteria |
| Paper # | NLC2021-15 |
| Date of Issue | 2021-09-09 (NLC) |
| Conference Information | |
| Committee | NLC |
|---|---|
| Conference Date | 2021/9/16(1days) |
| Place (in Japanese) | (See Japanese page) |
| Place (in English) | Online |
| Topics (in Japanese) | (See Japanese page) |
| Topics (in English) | The 18th Text Analytics Symposium |
| Chair | Kazutaka Shimada(Kyushu Inst. of Tech.) |
| Vice Chair | Mitsuo Yoshida(Univ. of Tsukuba) / Takeshi Kobayakawa(NHK) |
| Secretary | Mitsuo Yoshida(Univ. of Tokyo) / Takeshi Kobayakawa(Hiroshima Univ. of Economics) |
| Assistant | Kanjin Takahashi(Sansan) / Ko Mitsuda(NTT) |
| Paper Information | |
| Registration To | Technical Committee on Natural Language Understanding and Models of Communication |
|---|---|
| Language | JPN |
| Title (in Japanese) | (See Japanese page) |
| Sub Title (in Japanese) | (See Japanese page) |
| Title (in English) | Construction of Japanese Essay Data for Evaluating Automatic Essay Scoring System |
| Sub Title (in English) | |
| Keyword(1) | automatic essay scoring |
| Keyword(2) | Japanese native speakers |
| Keyword(3) | evaluation criteria |
| 1st Author's Name | Koichi Takeuchi |
| 1st Author's Affiliation | Okayama University(Okayama Univ.) |
| 2nd Author's Name | Masahiro Taguchi |
| 2nd Author's Affiliation | Okayama University(Okayama Univ.) |
| 3rd Author's Name | Yoshihiko Inada |
| 3rd Author's Affiliation | Okayama University(Okayama Univ.) |
| 4th Author's Name | Masaya Iizuka |
| 4th Author's Affiliation | Okayama University(Okayama Univ.) |
| 5th Author's Name | Tasuhiko Abo |
| 5th Author's Affiliation | Okayama University(Okayama Univ.) |
| 6th Author's Name | Hitoshi Ueda |
| 6th Author's Affiliation | Okayama University(Okayama Univ.) |
| Date | 2021-09-16 |
| Paper # | NLC2021-15 |
| Volume (vol) | vol.121 |
| Number (no) | NLC-178 |
| Page | pp.pp.40-44(NLC), |
| #Pages | 5 |
| Date of Issue | 2021-09-09 (NLC) |