| Presentation | 2023-01-29 Predictions and Attentions Acquired by Vision Transformer with Source-Target Attention from Dilated Convolutions on Small Data Sets Tatsuki Shimura, Katsumi Tadamura, Toshikazu Samura, |
|---|---|
| PDF Download Page |  PDF download Page Link |
| Abstract(in Japanese) | (See Japanese page) |
| Abstract(in English) | Vision Transformer (ViT) requires large data sets during pre-training phase to acquire high classification accuracy on any data sets. It has been proposed that ViT with convolutional input structure reduce the pre-training cost. In this study, we proposed ViT with source-target attention from dilated convolutions. We show that the proposed ViT acquire the same accuracy and attention as the conventional ViT trained with large data set even when the number of data is reduced in the pre-training phase. |
| Keyword(in Japanese) | (See Japanese page) |
| Keyword(in English) | Vision Transformer / Source-Target Attention / Dilated Convolution / Small data |
| Paper # | NLP2022-104,NC2022-88 |
| Date of Issue | 2023-01-21 (NLP, NC) |
| Conference Information | |
| Committee | NC / NLP |
|---|---|
| Conference Date | 2023/1/28(2days) |
| Place (in Japanese) | (See Japanese page) |
| Place (in English) | Future University Hakodate |
| Topics (in Japanese) | (See Japanese page) |
| Topics (in English) | NC, NLP, etc. |
| Chair | Hiroshi Yamakawa(Univ of Tokyo) / Akio Tsuneda(Kumamoto Univ.) |
| Vice Chair | Hirokazu Tanaka(Tokyo City Univ.) / Hiroyuki Torikai(Hosei Univ.) |
| Secretary | Hirokazu Tanaka(NTT) / Hiroyuki Torikai(NICT) |
| Assistant | Yoshimasa Tawatsuji(Waseda Univ.) / Tomoki Kurikawa(KMU) / Yuichi Yokoi(Nagasaki Univ.) / Yoshikazu Yamanaka(Utsunomiya Univ.) |
| Paper Information | |
| Registration To | Technical Committee on Neurocomputing / Technical Committee on Nonlinear Problems |
|---|---|
| Language | JPN |
| Title (in Japanese) | (See Japanese page) |
| Sub Title (in Japanese) | (See Japanese page) |
| Title (in English) | Predictions and Attentions Acquired by Vision Transformer with Source-Target Attention from Dilated Convolutions on Small Data Sets |
| Sub Title (in English) | |
| Keyword(1) | Vision Transformer |
| Keyword(2) | Source-Target Attention |
| Keyword(3) | Dilated Convolution |
| Keyword(4) | Small data |
| 1st Author's Name | Tatsuki Shimura |
| 1st Author's Affiliation | Yamaguchi University(Yamaguchi Univ) |
| 2nd Author's Name | Katsumi Tadamura |
| 2nd Author's Affiliation | Yamaguchi University(Yamaguchi Univ) |
| 3rd Author's Name | Toshikazu Samura |
| 3rd Author's Affiliation | Yamaguchi University(Yamaguchi Univ) |
| Date | 2023-01-29 |
| Paper # | NLP2022-104,NC2022-88 |
| Volume (vol) | vol.122 |
| Number (no) | NLP-373,NC-374 |
| Page | pp.pp.123-128(NLP), pp.123-128(NC), |
| #Pages | 6 |
| Date of Issue | 2023-01-21 (NLP, NC) |