| Presentation | 2019-06-18 ResNet and Batch-normalization Improve Data Separation Ability Yasutaka Furusho, Kazushi Ikeda, |
|---|---|
| PDF Download Page |  PDF download Page Link |
| Abstract(in Japanese) | (See Japanese page) |
| Abstract(in English) | The skip-connection and the batch-normalization (BN) in ResNet enable an extreme deep neural network to be trained with high performance. However, the reasons for its high performance are still unclear. A large ratio of the between-class distance to the within-class distance of feature vectors at the last hidden layer induces a high performance. Thus, we analyzed the change of these distances through hidden layers of the randomly initialized multilayer perceptron (MLP), the ResNet, and the ResNet with BN. Our results show that the MLP strongly decreases the between-class distance compared with the within-class distance and that the skip-connection and the BN relax this decrease of the between-class angle and improve the ratio of the distances. Moreover, the skip-connection and the BN relax the exponential decrease of the angle into the reciprocal decrease. We also analyzed the effects of training on the distances and show that the preservation of the angle through layers at initialization encourages trained neural networks to increase the ratio of the distances. Therefore, the skip-connection and the BN in the ResNet induce a high performance. |
| Keyword(in Japanese) | (See Japanese page) |
| Keyword(in English) | Deep neural network / ResNet / Skip-connection / Batch-normalization |
| Paper # | NC2019-18,IBISML2019-16 |
| Date of Issue | 2019-06-10 (NC, IBISML) |
| Conference Information | |
| Committee | NC / IBISML / IPSJ-MPS / IPSJ-BIO |
|---|---|
| Conference Date | 2019/6/17(3days) |
| Place (in Japanese) | (See Japanese page) |
| Place (in English) | Okinawa Institute of Science and Technology |
| Topics (in Japanese) | (See Japanese page) |
| Topics (in English) | Neurocomputing, Machine Learning Approach to Biodata Mining, and General |
| Chair | Hayaru Shouno(UEC) / Hisashi Kashima(Kyoto Univ.) / Masakazu Sekijima(Tokyo Tech) / Hiroyuki Kurata(Kyutech) |
| Vice Chair | Kazuyuki Samejima(Tamagawa Univ) / Masashi Sugiyama(Univ. of Tokyo) / Koji Tsuda(Univ. of Tokyo) |
| Secretary | Kazuyuki Samejima(NAIST) / Masashi Sugiyama(NTT) / Koji Tsuda(Nagoya Inst. of Tech.) / (AIST) / (Nagoya Univ.) |
| Assistant | Takashi Shinozaki(NICT) / Ken Takiyama(TUAT) / Tomoharu Iwata(NTT) / Shigeyuki Oba(Kyoto Univ.) |
| Paper Information | |
| Registration To | Technical Committee on Neurocomputing / Technical Committee on Infomation-Based Induction Sciences and Machine Learning / IPSJ Special Interest Group on Mathematical Modeling and Problem Solving / IPSJ Special Interest Group on Bioinformatics and Genomics |
|---|---|
| Language | JPN |
| Title (in Japanese) | (See Japanese page) |
| Sub Title (in Japanese) | (See Japanese page) |
| Title (in English) | ResNet and Batch-normalization Improve Data Separation Ability |
| Sub Title (in English) | |
| Keyword(1) | Deep neural network |
| Keyword(2) | ResNet |
| Keyword(3) | Skip-connection |
| Keyword(4) | Batch-normalization |
| 1st Author's Name | Yasutaka Furusho |
| 1st Author's Affiliation | Nara Institute of Science and Technology(NAIST) |
| 2nd Author's Name | Kazushi Ikeda |
| 2nd Author's Affiliation | Nara Institute of Science and Technology(NAIST) |
| Date | 2019-06-18 |
| Paper # | NC2019-18,IBISML2019-16 |
| Volume (vol) | vol.119 |
| Number (no) | NC-88,IBISML-89 |
| Page | pp.pp.81-86(NC), pp.103-108(IBISML), |
| #Pages | 6 |
| Date of Issue | 2019-06-10 (NC, IBISML) |