Achievement Award

Development and Commercialization of High-Speed Exploratory Video Retrieval

Jianquan LIU
Jianquan LIU
Satoshi YAMAZAKI
Satoshi YAMAZAKI
Youhei SASAKI
Youhei SASAKI

Exploratory video retrieval technology is a newly developed technology that can automatically discover individuals whose behavior differs from others without pre-registering them as search targets. This technology represents each individual in a video by a newly defined "Bio-motion Vector", which allows us to find individuals behaving differently from others without registration (Figure 1). Furthermore, with the newly developed "Hierarchical Vector Tree Structure", the discovery time is reduced to 1/10,000th for one million people. The Bio-motion Vector is composed of a Movement Vector, defined by the direction, speed, and time of movement, and a Bio-feature Vector that represents the individuality of a person. If the time variation of the movement vector is lowly correlated with others, it is judged that the individual's behavior differs from others. The Hierarchical Vector Tree Structure hierarchically classifies all Bio-motion Vectors based on the correlation between them (Figure 2), progressively reducing the number of vector comparisons due to the pruning effect of the tree structure. The conventional complexity of a brute force algorithm, O(N^2), is reduced to O(NlogN) by pruning, thereby speeding up the search. This technology can instantly find and present individuals behaving differently from others from continuously changing videos, making it effective for crowd control, crime prevention support, and searching for lost children. It has been commercialized and is being used in safety management at overseas facilities in five countries, including by police and airports, and in identifying performers within videos for TV companies.

This technology is an innovative foundational technology supporting multimedia applications such as text-based keyword search, speaker and spoken voice recognition and classification, and image search, and is also significant for exploring high-dimensional vectors, which are important in Generative AI technology. It has a tremendous ripple effect. It's academic superiority and advantages have been recognized through 48 papers presented at top international conferences in the field (2,3), 53 patent applications (27 granted in Japan, 19 granted overseas) and academic awards (4,5). Commercially, this technology was transferred into the video analysis AI software "NeoFace® Image Data Mining", in 2016. Moreover, it can be applied across a wide range of industries such as crowd control, crime prevention support, searching for lost children, marketing, and broadcast video editing, amounting to substantial societal significance. Furthermore, it has been recognized for its value with an Industrial Achievement Award and a Research and Engineering Award from the Information Processing Society of Japan in 2018 and 2020, respectively, a Technology Development Encouragement Award from the Japan Newspaper Publishers and Editors Association in 2018, an Invention Encouragement Award at the Kanto Regional Invention Awards in 2021, and the Minister of Education, Culture, Sports, Science and Technology Award -- the top award of the Electric Science and Technology Encouragement Awards in 2021. In light of the above, the awardees have indeed made remarkable achievements, and we trust this qualifies them for IEICE's Achievement Award.

Fig.1
Figure 1: (Left) Examples of behaviors differing from others, (Right) Visualization of behavioral correlation by proposed biometric motion vectors.
Fig.2
Figure 2: (Left) Concept of structuring vectors, (Right) Overview of the proposed hierarchical vector tree structure.

References

  1. J. Liu, et al. “A loitering discovery system using efficient similarity search based on similarity hierarchy”. IEICE Trans. on Fund., 100-A(2):367–375, 2017.
  2. J. Liu, et al. “AntiLoiter: A loitering discovery system for longtime video across multiple surveillance cameras”. ACM Conf. on Multimedia (ACM MM), 675-679, 2016.
  3. J. Liu, et al. “Visloiter: A system to visualize loiterers discovered from surveillance videos”. ACM SIGGRAPH Posters (SIGGRAPH), 47:1-47:2, 2016.
  4. Best Paper of the Year Award from the IEICE Technical Committee on Mathematical Systems Science and its Applications in 2015.
  5. Consecutive awards received at the Annual Conferences (DEIM) of the Database Society of Japan from 2016 to 2019 and 2023 (Best Demo Award in 2019, Excellent Paper Award in 2023), and Best BNI Paper Award at ACM MM 2022.
  6. Patent No. 6183376, US 10713229, EP 2945071 and others in DE, FR, GB, etc.