Paper Abstract and Keywords |
Presentation |
2016-11-16 15:00
Statistical Mechanical Analysis of Fast Online Learning with Weight Normalization Yuki Yoshida, Ryo Karakida, Masato Okada (UTokyo), Shun-ichi Amari (RIKEN) IBISML2016-60 |
Abstract |
(in Japanese) |
(See Japanese page) |
(in English) |
Weight normalization (WN), a newly developed optimization algorithm for neural networks by Salimans & Kingma(2016), factorizes the weight vector of a neural network into a radial length and a direction vector, and the factorized parameters follow their steepest gradient descent update. They showed that learning with WN yields better converging speed in several practical tasks including image recognition and reinforcement learning than learning with the conventional steepest descent. However, it remains theoretically unclear why this method works well. In this study, we used a statistical mechanical approach to analyze on-line learning in single layer linear and nonlinear perceptrons with WN. By deriving order parameters of the dynamics of learning, we confirmed quantitatively that WN achieves fast converging speed by automatically tuning the effective learning rate, irrespective of the nonlinearity of the neural network. This fast converging is realized when the initial value of the radial length is near the global minimum; therefore, our theory suggests that it is important to choose the initial value of the radial length appropriately when using WN. |
Keyword |
(in Japanese) |
(See Japanese page) |
(in English) |
Neural network / Weight normalization / Online learning / Statistical mechanics / / / / |
Reference Info. |
IEICE Tech. Rep., vol. 116, no. 300, IBISML2016-60, pp. 101-108, Nov. 2016. |
Paper # |
IBISML2016-60 |
Date of Issue |
2016-11-09 (IBISML) |
ISSN |
Print edition: ISSN 0913-5685 Online edition: ISSN 2432-6380 |
Copyright and reproduction |
All rights are reserved and no part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording, or any information storage and retrieval system, without permission in writing from the publisher. Notwithstanding, instructors are permitted to photocopy isolated articles for noncommercial classroom use without fee. (License No.: 10GA0019/12GB0052/13GB0056/17GB0034/18GB0034) |
Download PDF |
IBISML2016-60 |
Conference Information |
Committee |
IBISML |
Conference Date |
2016-11-16 - 2016-11-18 |
Place (in Japanese) |
(See Japanese page) |
Place (in English) |
Kyoto Univ. |
Topics (in Japanese) |
(See Japanese page) |
Topics (in English) |
Information-Based Induction Science Workshop (IBIS2016) |
Paper Information |
Registration To |
IBISML |
Conference Code |
2016-11-IBISML |
Language |
Japanese |
Title (in Japanese) |
(See Japanese page) |
Sub Title (in Japanese) |
(See Japanese page) |
Title (in English) |
Statistical Mechanical Analysis of Fast Online Learning with Weight Normalization |
Sub Title (in English) |
|
Keyword(1) |
Neural network |
Keyword(2) |
Weight normalization |
Keyword(3) |
Online learning |
Keyword(4) |
Statistical mechanics |
Keyword(5) |
|
Keyword(6) |
|
Keyword(7) |
|
Keyword(8) |
|
1st Author's Name |
Yuki Yoshida |
1st Author's Affiliation |
The University of Tokyo (UTokyo) |
2nd Author's Name |
Ryo Karakida |
2nd Author's Affiliation |
The University of Tokyo (UTokyo) |
3rd Author's Name |
Masato Okada |
3rd Author's Affiliation |
The University of Tokyo (UTokyo) |
4th Author's Name |
Shun-ichi Amari |
4th Author's Affiliation |
RIKEN (RIKEN) |
5th Author's Name |
|
5th Author's Affiliation |
() |
6th Author's Name |
|
6th Author's Affiliation |
() |
7th Author's Name |
|
7th Author's Affiliation |
() |
8th Author's Name |
|
8th Author's Affiliation |
() |
9th Author's Name |
|
9th Author's Affiliation |
() |
10th Author's Name |
|
10th Author's Affiliation |
() |
11th Author's Name |
|
11th Author's Affiliation |
() |
12th Author's Name |
|
12th Author's Affiliation |
() |
13th Author's Name |
|
13th Author's Affiliation |
() |
14th Author's Name |
|
14th Author's Affiliation |
() |
15th Author's Name |
|
15th Author's Affiliation |
() |
16th Author's Name |
|
16th Author's Affiliation |
() |
17th Author's Name |
|
17th Author's Affiliation |
() |
18th Author's Name |
|
18th Author's Affiliation |
() |
19th Author's Name |
|
19th Author's Affiliation |
() |
20th Author's Name |
|
20th Author's Affiliation |
() |
Speaker |
Author-1 |
Date Time |
2016-11-16 15:00:00 |
Presentation Time |
180 minutes |
Registration for |
IBISML |
Paper # |
IBISML2016-60 |
Volume (vol) |
vol.116 |
Number (no) |
no.300 |
Page |
pp.101-108 |
#Pages |
8 |
Date of Issue |
2016-11-09 (IBISML) |
|