Presentation | 2003/7/21 Plateau and Quasi-Plateau Due to Singularity in Multilayer Perceptrons : Standard Gradient vs. Natural Gradient Hyeyoung PARK, Masato INOUE, Masato OKADA, |
---|---|
PDF Download Page | PDF download Page Link |
Abstract(in Japanese) | (See Japanese page) |
Abstract(in English) | In a hierarchical set of models, a low order model is included in the parameter space of a larger order model as a union of subspaces, which makes singularities in the parameter space. The complex singular structure is known to affect estimation performances and learning dynamics of the models. Although there have been a number of studies on properties of obtained estimators for the models, there are few studies on the dynamical properties of learning exploited for obtaining the estimators. Taking Statistical mechanical approach, we investigate influences of singularities on dynamics of standard gradient learning and natural gradient learning under various learning conditions for mutilayer percpetorns. In the standard gradient learning, we found a quasi-plateau phenomenon, which is severer than the well known plateau in some cases. The slow convergence due to the quasi-plateau and plateau becomes extremely serious when an optimal point is in a neighborhood of singularity. In the natural gradient learning, however, the quasi-plateau and plateau are not observed and convergence speed is hardly affected by singularity. |
Keyword(in Japanese) | (See Japanese page) |
Keyword(in English) | Multilayer Perceptron / Unidentifiable parameter / Singular structures / On-line learning dyanamics / Statistical mechanics |
Paper # | NC2003-24 |
Date of Issue |
Conference Information | |
Committee | NC |
---|---|
Conference Date | 2003/7/21(1days) |
Place (in Japanese) | (See Japanese page) |
Place (in English) | |
Topics (in Japanese) | (See Japanese page) |
Topics (in English) | |
Chair | |
Vice Chair | |
Secretary | |
Assistant |
Paper Information | |
Registration To | Neurocomputing (NC) |
---|---|
Language | ENG |
Title (in Japanese) | (See Japanese page) |
Sub Title (in Japanese) | (See Japanese page) |
Title (in English) | Plateau and Quasi-Plateau Due to Singularity in Multilayer Perceptrons : Standard Gradient vs. Natural Gradient |
Sub Title (in English) | |
Keyword(1) | Multilayer Perceptron |
Keyword(2) | Unidentifiable parameter |
Keyword(3) | Singular structures |
Keyword(4) | On-line learning dyanamics |
Keyword(5) | Statistical mechanics |
1st Author's Name | Hyeyoung PARK |
1st Author's Affiliation | RIKEN Brain Science Institute() |
2nd Author's Name | Masato INOUE |
2nd Author's Affiliation | RIKEN Brain Science Institute |
3rd Author's Name | Masato OKADA |
3rd Author's Affiliation | RIKEN Brain Science Institute:"Intelligent Cooperation and Control", PRESTO, JST |
Date | 2003/7/21 |
Paper # | NC2003-24 |
Volume (vol) | vol.103 |
Number (no) | 227 |
Page | pp.pp.- |
#Pages | 6 |
Date of Issue |