| Presentation | 2019-06-18 Additive or Concatenating Skip-connections Overcome the Degradation Problem of the Classic Feedforward Neural Network Yasutaka Furusho, Kazushi Ikeda, |
|---|---|
| PDF Download Page |  PDF download Page Link |
| Abstract(in Japanese) | (See Japanese page) |
| Abstract(in English) | The classic feedforward neural networks like the multilayer perceptron (MLP) degrades its empirical risk by training even though it stacks more layers. To overcome this problem, the ResNet which has additive skip-connections and the DenseNet which has concatenating skip-connections were proposed. These skip-connections enable an extreme deep neural network (DNN) to be trained with high performance. However, the reasons for these successes and when to prefer the one skip-connection to the other are unclear. A large ratio of the between-class distance to the within-class distance of feature vectors at the last hidden layer induces a high performance. Thus, we analyzed the change of these distances through hidden layers of the randomly initialized MLP, the ResNet, and the DenseNet. Our results show that the MLP strongly decreases the between-class distance compared with the within-class distance and that both skip-connections relax this decrease of the between-class angle and improve the ratio of the distances. In particular, the concatenating skip-connection is more preferable to the additive skip-connection if a DNN is extremely deep. Moreover, the additive skip-connection relax the exponential decrease of the angle into the sub-exponential decrease and the concatenating skip-connection relax this decrease into the reciprocal decrease. We also analyzed the effects of training on the distances and show that the preservation of the angle through layers at initialization encourages trained neural networks to increase the ratio of the distances. Therefore, both skip-connections induce high performance. |
| Keyword(in Japanese) | (See Japanese page) |
| Keyword(in English) | Deep neural network / ResNet / DenseNet / Skip-connection |
| Paper # | NC2019-17,IBISML2019-15 |
| Date of Issue | 2019-06-10 (NC, IBISML) |
| Conference Information | |
| Committee | NC / IBISML / IPSJ-MPS / IPSJ-BIO |
|---|---|
| Conference Date | 2019/6/17(3days) |
| Place (in Japanese) | (See Japanese page) |
| Place (in English) | Okinawa Institute of Science and Technology |
| Topics (in Japanese) | (See Japanese page) |
| Topics (in English) | Neurocomputing, Machine Learning Approach to Biodata Mining, and General |
| Chair | Hayaru Shouno(UEC) / Hisashi Kashima(Kyoto Univ.) / Masakazu Sekijima(Tokyo Tech) / Hiroyuki Kurata(Kyutech) |
| Vice Chair | Kazuyuki Samejima(Tamagawa Univ) / Masashi Sugiyama(Univ. of Tokyo) / Koji Tsuda(Univ. of Tokyo) |
| Secretary | Kazuyuki Samejima(NAIST) / Masashi Sugiyama(NTT) / Koji Tsuda(Nagoya Inst. of Tech.) / (AIST) / (Nagoya Univ.) |
| Assistant | Takashi Shinozaki(NICT) / Ken Takiyama(TUAT) / Tomoharu Iwata(NTT) / Shigeyuki Oba(Kyoto Univ.) |
| Paper Information | |
| Registration To | Technical Committee on Neurocomputing / Technical Committee on Infomation-Based Induction Sciences and Machine Learning / IPSJ Special Interest Group on Mathematical Modeling and Problem Solving / IPSJ Special Interest Group on Bioinformatics and Genomics |
|---|---|
| Language | JPN |
| Title (in Japanese) | (See Japanese page) |
| Sub Title (in Japanese) | (See Japanese page) |
| Title (in English) | Additive or Concatenating Skip-connections Overcome the Degradation Problem of the Classic Feedforward Neural Network |
| Sub Title (in English) | |
| Keyword(1) | Deep neural network |
| Keyword(2) | ResNet |
| Keyword(3) | DenseNet |
| Keyword(4) | Skip-connection |
| 1st Author's Name | Yasutaka Furusho |
| 1st Author's Affiliation | Nara Institute of Science and Technology(NAIST) |
| 2nd Author's Name | Kazushi Ikeda |
| 2nd Author's Affiliation | Nara Institute of Science and Technology(NAIST) |
| Date | 2019-06-18 |
| Paper # | NC2019-17,IBISML2019-15 |
| Volume (vol) | vol.119 |
| Number (no) | NC-88,IBISML-89 |
| Page | pp.pp.75-80(NC), pp.97-102(IBISML), |
| #Pages | 6 |
| Date of Issue | 2019-06-10 (NC, IBISML) |