| Presentation | 2020-10-21 [Invited Talk] Understanding deep learning via differential equations Sho Sonoda, |
|---|---|
| PDF Download Page |  PDF download Page Link |
| Abstract(in Japanese) | (See Japanese page) |
| Abstract(in English) | The goal of this study is to understand the information processing mechanism in a deep neural network (DNN) as a curve $f:[0,1]times Z to Z$ in a latent feature space $Z$ or the time evolution of the distribution $p(t,z)$ of latent features on $Z$. Although real DNNs may have an arbitrary freely connected network, we consider a deep network where multiple hidden layers are cascaded. A cascaded network can be identified with a composite of maps $f_L circ cdots circ f_1:Z^1 to cdots to Z^L$, and its information processing can be identified with a sequence ${ z_ell }_{ell in NN}$ or a curve $z(t)$ in a common latent feature space $Z$. Hence by specifying the generation process of sequences or curves, we can understand and hopefully improve the information processing mechanism in the DNN. This conversion approach, namely, from abstract information processing mechanisms to more concrete sequences or curves, is now known as the Neural ODE (or ODE-Net), and draws much attention in the field of deep learning study. In this talk, based on Sonoda and Murata (2019) ``Transport Analysis of Infinitely Deep Neural Network'' JMLR, the presenter shows that for the denoising autoencoder (DAE), the feature distribution $p(t,z)$ develops according to the backward heat equation. In addition, if time permits, he introduces a new Radon-type reconstruction formula for ReLU nets, and consider a new differential equation for ReLU nets based on the reconstruction formula. |
| Keyword(in Japanese) | (See Japanese page) |
| Keyword(in English) | deep neural network / optimal control / autoencoder / backward heat equation / Radon-type reconstruction formula |
| Paper # | IBISML2020-24 |
| Date of Issue | 2020-10-13 (IBISML) |
| Conference Information | |
| Committee | IBISML |
|---|---|
| Conference Date | 2020/10/20(3days) |
| Place (in Japanese) | (See Japanese page) |
| Place (in English) | Online |
| Topics (in Japanese) | (See Japanese page) |
| Topics (in English) | Organized Sessions on Frontiers of Machine Learning and General Sessions |
| Chair | Ichiro Takeuchi(Nagoya Inst. of Tech.) |
| Vice Chair | Masashi Sugiyama(Univ. of Tokyo) / Koji Tsuda(Univ. of Tokyo) |
| Secretary | Masashi Sugiyama(AIST) / Koji Tsuda(NTT) |
| Assistant | Atsuyoshi Nakamura(Hokkaido Univ.) / Shigeyuki Oba(Kyoto Univ.) |
| Paper Information | |
| Registration To | Technical Committee on Infomation-Based Induction Sciences and Machine Learning |
|---|---|
| Language | JPN-ONLY |
| Title (in Japanese) | (See Japanese page) |
| Sub Title (in Japanese) | (See Japanese page) |
| Title (in English) | [Invited Talk] Understanding deep learning via differential equations |
| Sub Title (in English) | |
| Keyword(1) | deep neural network |
| Keyword(2) | optimal control |
| Keyword(3) | autoencoder |
| Keyword(4) | backward heat equation |
| Keyword(5) | Radon-type reconstruction formula |
| 1st Author's Name | Sho Sonoda |
| 1st Author's Affiliation | RIKEN(RIKEN) |
| Date | 2020-10-21 |
| Paper # | IBISML2020-24 |
| Volume (vol) | vol.120 |
| Number (no) | IBISML-195 |
| Page | pp.pp.42-42(IBISML), |
| #Pages | 1 |
| Date of Issue | 2020-10-13 (IBISML) |