| Presentation | 2021-01-25 Efficient Attention Mechanism by Softmax Function with Trained Coefficient Kaito Hirota, O'uchi Shinichi, Fujita Masahiro, |
|---|---|
| PDF Download Page |  PDF download Page Link |
| Abstract(in Japanese) | (See Japanese page) |
| Abstract(in English) | BERT is a neural network model which has accomplished state-of-the-art performance on eleven natural language processing tasks such as inference and paraphrasing. So it is desired to make BERT based computations available on edge devices. We propose an efficient hardware implementation method for the part of this model by modifying Softmax function. Softmax function is a part of the most significant calculation in BERT in terms of computation cost, and its hardware implementation on FPGA or ASIC has not been explored. We have succeeded in reducing the number of additions and exponential functions to 1%, while retaining 95% of the BERT’s accuracy through experiments. |
| Keyword(in Japanese) | (See Japanese page) |
| Keyword(in English) | Deep Learning / Natural Language Processing / BERT / Attention / Hardware Design / FPGA / Softmax Function |
| Paper # | VLD2020-48,CPSY2020-31,RECONF2020-67 |
| Date of Issue | 2021-01-18 (VLD, CPSY, RECONF) |
| Conference Information | |
| Committee | CPSY / RECONF / VLD / IPSJ-ARC / IPSJ-SLDM |
|---|---|
| Conference Date | 2021/1/25(2days) |
| Place (in Japanese) | (See Japanese page) |
| Place (in English) | Online |
| Topics (in Japanese) | (See Japanese page) |
| Topics (in English) | FPGA Applications, etc. |
| Chair | Hidetsugu Irie(Univ. of Tokyo) / Yuichiro Shibata(Nagasaki Univ.) / Daisuke Fukuda(Fujitsu Labs.) / Hiroshi Inoue(Kyushu Univ.) / Yuichi Nakamura(NEC) |
| Vice Chair | Michihiro Koibuchi(NII) / Kota Nakajima(Fujitsu Lab.) / Kentaro Sano(RIKEN) / Yoshiki Yamaguchi(Tsukuba Univ.) / Kazutoshi Kobayashi(Kyoto Inst. of Tech.) |
| Secretary | Michihiro Koibuchi(Hokkaido Univ.) / Kota Nakajima(Nagoya Inst. of Tech.) / Kentaro Sano(e-trees.Japan) / Yoshiki Yamaguchi(NEC) / Kazutoshi Kobayashi(Hitachi) / (Osaka Univ.) / (Fujitsu lab.) |
| Assistant | Shugo Ogawa(Hitachi) / Eiji Arima(Univ. of Tokyo) / Hiroki Nakahara(Tokyo Inst. of Tech.) / Yukitaka Takemura(INTEL) / Takuma Nishimoto(Hitachi) |
| Paper Information | |
| Registration To | Technical Committee on Computer Systems / Technical Committee on Reconfigurable Systems / Technical Committee on VLSI Design Technologies / Special Interest Group on System Architecture / Special Interest Group on System and LSI Design Methodology |
|---|---|
| Language | JPN |
| Title (in Japanese) | (See Japanese page) |
| Sub Title (in Japanese) | (See Japanese page) |
| Title (in English) | Efficient Attention Mechanism by Softmax Function with Trained Coefficient |
| Sub Title (in English) | |
| Keyword(1) | Deep Learning |
| Keyword(2) | Natural Language Processing |
| Keyword(3) | BERT |
| Keyword(4) | Attention |
| Keyword(5) | Hardware Design |
| Keyword(6) | FPGA |
| Keyword(7) | Softmax Function |
| 1st Author's Name | Kaito Hirota |
| 1st Author's Affiliation | the University of Tokyo(UT) |
| 2nd Author's Name | O'uchi Shinichi |
| 2nd Author's Affiliation | National Institute of Advanced Industrial Science and Technology(AIST) |
| 3rd Author's Name | Fujita Masahiro |
| 3rd Author's Affiliation | the University of Tokyo(UT) |
| Date | 2021-01-25 |
| Paper # | VLD2020-48,CPSY2020-31,RECONF2020-67 |
| Volume (vol) | vol.120 |
| Number (no) | VLD-337,CPSY-338,RECONF-339 |
| Page | pp.pp.52-57(VLD), pp.52-57(CPSY), pp.52-57(RECONF), |
| #Pages | 6 |
| Date of Issue | 2021-01-18 (VLD, CPSY, RECONF) |