Presentation 2003/7/21
Plateau and Quasi-Plateau Due to Singularity in Multilayer Perceptrons : Standard Gradient vs. Natural Gradient
Hyeyoung PARK, Masato INOUE, Masato OKADA,
PDF Download Page PDF download Page Link
Abstract(in Japanese) (See Japanese page)
Abstract(in English) In a hierarchical set of models, a low order model is included in the parameter space of a larger order model as a union of subspaces, which makes singularities in the parameter space. The complex singular structure is known to affect estimation performances and learning dynamics of the models. Although there have been a number of studies on properties of obtained estimators for the models, there are few studies on the dynamical properties of learning exploited for obtaining the estimators. Taking Statistical mechanical approach, we investigate influences of singularities on dynamics of standard gradient learning and natural gradient learning under various learning conditions for mutilayer percpetorns. In the standard gradient learning, we found a quasi-plateau phenomenon, which is severer than the well known plateau in some cases. The slow convergence due to the quasi-plateau and plateau becomes extremely serious when an optimal point is in a neighborhood of singularity. In the natural gradient learning, however, the quasi-plateau and plateau are not observed and convergence speed is hardly affected by singularity.
Keyword(in Japanese) (See Japanese page)
Keyword(in English) Multilayer Perceptron / Unidentifiable parameter / Singular structures / On-line learning dyanamics / Statistical mechanics
Paper # NC2003-24
Date of Issue

Conference Information
Committee NC
Conference Date 2003/7/21(1days)
Place (in Japanese) (See Japanese page)
Place (in English)
Topics (in Japanese) (See Japanese page)
Topics (in English)
Chair
Vice Chair
Secretary
Assistant

Paper Information
Registration To Neurocomputing (NC)
Language ENG
Title (in Japanese) (See Japanese page)
Sub Title (in Japanese) (See Japanese page)
Title (in English) Plateau and Quasi-Plateau Due to Singularity in Multilayer Perceptrons : Standard Gradient vs. Natural Gradient
Sub Title (in English)
Keyword(1) Multilayer Perceptron
Keyword(2) Unidentifiable parameter
Keyword(3) Singular structures
Keyword(4) On-line learning dyanamics
Keyword(5) Statistical mechanics
1st Author's Name Hyeyoung PARK
1st Author's Affiliation RIKEN Brain Science Institute()
2nd Author's Name Masato INOUE
2nd Author's Affiliation RIKEN Brain Science Institute
3rd Author's Name Masato OKADA
3rd Author's Affiliation RIKEN Brain Science Institute:"Intelligent Cooperation and Control", PRESTO, JST
Date 2003/7/21
Paper # NC2003-24
Volume (vol) vol.103
Number (no) 227
Page pp.pp.-
#Pages 6
Date of Issue