| Presentation | 2024-01-25 Estimation of 3D Fingertips Coordinates Using Contrastive Embeddings from Hand Images Tatsuya Abe, Takeshi Umezawa, Noritaka Osawa, |
|---|---|
| PDF Download Page |  PDF download Page Link |
| Abstract(in Japanese) | (See Japanese page) |
| Abstract(in English) | This study evaluated a method for estimating the 3D coordinates of fingertips from hand images when manipulating objects in a virtual/mixed reality space. Deep learning is effective for estimating the 3D coordinates of fingertips from hand images, but a large amount of data collection is required to construct an estimation model with high accuracy and generalization performance. We compared an estimation model, which is based on a pre-trained model using contrastive learning and fine-tuned with supervised learning using a small amount of hand images, with an estimation model constructed using supervised learning without contrastive learning and evaluated the estimation accuracy of each. We adopted SimCLR as the architecture for contrastive learning and performed representation learning by augmenting the original data with random transformations of image size and contrast. We used the performance of estimation from palm images which include fingertips as a baseline , and compared it with the performance of estimation from images of the back of hand where the fingertips are hidden. From the results, we have clarified the potential for improving estimation accuracy using a model that employs representation learning, as well as the challenges to be addressed in the future. |
| Keyword(in Japanese) | (See Japanese page) |
| Keyword(in English) | 3D User Interface / Image Recognition / Contrastive Learning / Deep Learning / Coordinate Estimation |
| Paper # | PRMU2023-40 |
| Date of Issue | 2024-01-18 (PRMU) |
| Conference Information | |
| Committee | PRMU / MVE / VRSJ-SIG-MR / IPSJ-CVIM |
|---|---|
| Conference Date | 2024/1/25(2days) |
| Place (in Japanese) | (See Japanese page) |
| Place (in English) | Keio Univ. (Hiyoshi Campus) |
| Topics (in Japanese) | (See Japanese page) |
| Topics (in English) | |
| Chair | Kunio Kashio(NTT) / Kiyoshi Kiyokawa(NAIST) / / 日浦 慎作(兵庫県立大) |
| Vice Chair | Takuya Funatomi(NAIST) / Go Irie(Tokyo Univ. of Science) / Sumaru Niida(KDDI Research) |
| Secretary | Takuya Funatomi(Tokyo Inst. of Tech.) / Go Irie(Riken) / Sumaru Niida(Otsuma Women's University) / (DNP) / (Kyushu Univ.) |
| Assistant | Kei Shimonishi(Kyoto Univ.) / Kensho Hara(AIST) / Hidehiko Shishido(Soka University) / Atsushi Nakazawa(Kyoto Univ.) / Naoya Tojo(KDDI Research) / Naoki Hagiyama(NTT) / Yuji Tatada(Univ. of Tokyo) |
| Paper Information | |
| Registration To | Technical Committee on Pattern Recognition and Media Understanding / Technical Committee on Media Experience and Virtual Environment / SIG-MR / Special Interest Group on Computer Vision and Image Media |
|---|---|
| Language | JPN |
| Title (in Japanese) | (See Japanese page) |
| Sub Title (in Japanese) | (See Japanese page) |
| Title (in English) | Estimation of 3D Fingertips Coordinates Using Contrastive Embeddings from Hand Images |
| Sub Title (in English) | |
| Keyword(1) | 3D User Interface |
| Keyword(2) | Image Recognition |
| Keyword(3) | Contrastive Learning |
| Keyword(4) | Deep Learning |
| Keyword(5) | Coordinate Estimation |
| 1st Author's Name | Tatsuya Abe |
| 1st Author's Affiliation | Chiba University(Chiba Univ.) |
| 2nd Author's Name | Takeshi Umezawa |
| 2nd Author's Affiliation | Chiba University(Chiba Univ.) |
| 3rd Author's Name | Noritaka Osawa |
| 3rd Author's Affiliation | Chiba University(Chiba Univ.) |
| Date | 2024-01-25 |
| Paper # | PRMU2023-40 |
| Volume (vol) | vol.123 |
| Number (no) | PRMU-358 |
| Page | pp.pp.7-12(PRMU), |
| #Pages | 6 |
| Date of Issue | 2024-01-18 (PRMU) |