| Presentation | 2002/10/10 Natural Gradient Descent is not Affected by the Correlation of Hidden Layer Units in Multilaver Percentrons Masato INOUE, Hyeyoung PARK, Masato OKADA, |
|---|---|
| PDF Download Page | PDF download Page Link |
| Abstract(in Japanese) | (See Japanese page) |
| Abstract(in English) | The permutation symmetry of the hidden units in multilayer perceptrons causes the saddle structure and plateaus of the learning dynamics in gradient learning methods. The correlation of the weight vectors in the teacher network is supposed to affect this saddle structure resulting in the prolonged learning time, but this mechanism is still unclear. In this paper, we discuss it with regard to the soft committee machines and the on-line learning using statistical mechanics. Conventional steepest gradient descent needs longer time depending on the correlation of the weight vectors. On the other hand, natural gradient descent has no plateaus in the limit of the small learning rate even though the weight vectors have the strong correlation, which worsen the singularity of the Fisher information matrix. Analytical results supports these dynamics around the saddle point. |
| Keyword(in Japanese) | (See Japanese page) |
| Keyword(in English) | Natural gradient descent / perceptron / soft committee machine / singularity / saddle / plateau |
| Paper # | NC2002-54 |
| Date of Issue |
| Conference Information | |
| Committee | NC |
|---|---|
| Conference Date | 2002/10/10(1days) |
| Place (in Japanese) | (See Japanese page) |
| Place (in English) | |
| Topics (in Japanese) | (See Japanese page) |
| Topics (in English) | |
| Chair | |
| Vice Chair | |
| Secretary | |
| Assistant | |
| Paper Information | |
| Registration To | Neurocomputing (NC) |
|---|---|
| Language | JPN |
| Title (in Japanese) | (See Japanese page) |
| Sub Title (in Japanese) | (See Japanese page) |
| Title (in English) | Natural Gradient Descent is not Affected by the Correlation of Hidden Layer Units in Multilaver Percentrons |
| Sub Title (in English) | |
| Keyword(1) | Natural gradient descent |
| Keyword(2) | perceptron |
| Keyword(3) | soft committee machine |
| Keyword(4) | singularity |
| Keyword(5) | saddle |
| Keyword(6) | plateau |
| 1st Author's Name | Masato INOUE |
| 1st Author's Affiliation | Lab. for Mathematical Neuroscience, Brain Science Institute, RIKEN:Dept. of Otolaiyngology-Head and Neck Surgery, Graduate School of Medicine, Kyoto University() |
| 2nd Author's Name | Hyeyoung PARK |
| 2nd Author's Affiliation | Lab. for Mathematical Neuroscience, Brain Science Institute, RIKEN |
| 3rd Author's Name | Masato OKADA |
| 3rd Author's Affiliation | Lab. for Mathematical Neuroscience, Brain Science Institute, RIKEN |
| Date | 2002/10/10 |
| Paper # | NC2002-54 |
| Volume (vol) | vol.102 |
| Number (no) | 381 |
| Page | pp.pp.- |
| #Pages | 6 |
| Date of Issue | |