Presentation 2002/10/10
Natural Gradient Descent is not Affected by the Correlation of Hidden Layer Units in Multilaver Percentrons
Masato INOUE, Hyeyoung PARK, Masato OKADA,
PDF Download Page PDF download Page Link
Abstract(in Japanese) (See Japanese page)
Abstract(in English) The permutation symmetry of the hidden units in multilayer perceptrons causes the saddle structure and plateaus of the learning dynamics in gradient learning methods. The correlation of the weight vectors in the teacher network is supposed to affect this saddle structure resulting in the prolonged learning time, but this mechanism is still unclear. In this paper, we discuss it with regard to the soft committee machines and the on-line learning using statistical mechanics. Conventional steepest gradient descent needs longer time depending on the correlation of the weight vectors. On the other hand, natural gradient descent has no plateaus in the limit of the small learning rate even though the weight vectors have the strong correlation, which worsen the singularity of the Fisher information matrix. Analytical results supports these dynamics around the saddle point.
Keyword(in Japanese) (See Japanese page)
Keyword(in English) Natural gradient descent / perceptron / soft committee machine / singularity / saddle / plateau
Paper # NC2002-54
Date of Issue

Conference Information
Committee NC
Conference Date 2002/10/10(1days)
Place (in Japanese) (See Japanese page)
Place (in English)
Topics (in Japanese) (See Japanese page)
Topics (in English)
Chair
Vice Chair
Secretary
Assistant

Paper Information
Registration To Neurocomputing (NC)
Language JPN
Title (in Japanese) (See Japanese page)
Sub Title (in Japanese) (See Japanese page)
Title (in English) Natural Gradient Descent is not Affected by the Correlation of Hidden Layer Units in Multilaver Percentrons
Sub Title (in English)
Keyword(1) Natural gradient descent
Keyword(2) perceptron
Keyword(3) soft committee machine
Keyword(4) singularity
Keyword(5) saddle
Keyword(6) plateau
1st Author's Name Masato INOUE
1st Author's Affiliation Lab. for Mathematical Neuroscience, Brain Science Institute, RIKEN:Dept. of Otolaiyngology-Head and Neck Surgery, Graduate School of Medicine, Kyoto University()
2nd Author's Name Hyeyoung PARK
2nd Author's Affiliation Lab. for Mathematical Neuroscience, Brain Science Institute, RIKEN
3rd Author's Name Masato OKADA
3rd Author's Affiliation Lab. for Mathematical Neuroscience, Brain Science Institute, RIKEN
Date 2002/10/10
Paper # NC2002-54
Volume (vol) vol.102
Number (no) 381
Page pp.pp.-
#Pages 6
Date of Issue