| Presentation | 2019-06-18 Theoretical Analysis of the Fixup Initialization for Fast Convergence and High Generalization Ability Yasutaka Furusho, Kazushi Ikeda, |
|---|---|
| PDF Download Page |  PDF download Page Link |
| Abstract(in Japanese) | (See Japanese page) |
| Abstract(in English) | The Fixup initialization is a new initialization method of ResNet for a fast convergence with a high learning rate of SGD and a high generalization ability. However, the reasons for its high performance are not clear. Both the learning rate and the generalization ability are affected by the loss landscape at initialization, that is, the maximum eigenvalue of the Hessian matrix. Thus, we calculated the maximum eigenvalue of the ResNet and found that the maximum eigenvalue with the Fixup initialization has at most the square root order with respect to the depth of the ResNet while that with the He initialization has the exponential order. This small eigenvalue leads to a fast convergence with a high learning rate of SGD and a high generalization ability. |
| Keyword(in Japanese) | (See Japanese page) |
| Keyword(in English) | Deep neural network / ResNet / Initialization / Fixup |
| Paper # | NC2019-19,IBISML2019-17 |
| Date of Issue | 2019-06-10 (NC, IBISML) |
| Conference Information | |
| Committee | NC / IBISML / IPSJ-MPS / IPSJ-BIO |
|---|---|
| Conference Date | 2019/6/17(3days) |
| Place (in Japanese) | (See Japanese page) |
| Place (in English) | Okinawa Institute of Science and Technology |
| Topics (in Japanese) | (See Japanese page) |
| Topics (in English) | Neurocomputing, Machine Learning Approach to Biodata Mining, and General |
| Chair | Hayaru Shouno(UEC) / Hisashi Kashima(Kyoto Univ.) / Masakazu Sekijima(Tokyo Tech) / Hiroyuki Kurata(Kyutech) |
| Vice Chair | Kazuyuki Samejima(Tamagawa Univ) / Masashi Sugiyama(Univ. of Tokyo) / Koji Tsuda(Univ. of Tokyo) |
| Secretary | Kazuyuki Samejima(NAIST) / Masashi Sugiyama(NTT) / Koji Tsuda(Nagoya Inst. of Tech.) / (AIST) / (Nagoya Univ.) |
| Assistant | Takashi Shinozaki(NICT) / Ken Takiyama(TUAT) / Tomoharu Iwata(NTT) / Shigeyuki Oba(Kyoto Univ.) |
| Paper Information | |
| Registration To | Technical Committee on Neurocomputing / Technical Committee on Infomation-Based Induction Sciences and Machine Learning / IPSJ Special Interest Group on Mathematical Modeling and Problem Solving / IPSJ Special Interest Group on Bioinformatics and Genomics |
|---|---|
| Language | JPN |
| Title (in Japanese) | (See Japanese page) |
| Sub Title (in Japanese) | (See Japanese page) |
| Title (in English) | Theoretical Analysis of the Fixup Initialization for Fast Convergence and High Generalization Ability |
| Sub Title (in English) | |
| Keyword(1) | Deep neural network |
| Keyword(2) | ResNet |
| Keyword(3) | Initialization |
| Keyword(4) | Fixup |
| 1st Author's Name | Yasutaka Furusho |
| 1st Author's Affiliation | Nara Institute of Science and Technology(NAIST) |
| 2nd Author's Name | Kazushi Ikeda |
| 2nd Author's Affiliation | Nara Institute of Science and Technology(NAIST) |
| Date | 2019-06-18 |
| Paper # | NC2019-19,IBISML2019-17 |
| Volume (vol) | vol.119 |
| Number (no) | NC-88,IBISML-89 |
| Page | pp.pp.87-92(NC), pp.109-114(IBISML), |
| #Pages | 6 |
| Date of Issue | 2019-06-10 (NC, IBISML) |