Theme 1 from KDDI

Analysis on Route Information Failure in IP Core Networks
by NFV-Based Test Environment

Theme 2 from NEC

Network State Estimation
by Analyzing Raw Video Data

Theme 3 from RISING
(Not for Final Conference of ITU)

WiFi Location Estimation

Overview

Artificial Intelligence (AI) will be the dominant technology of the future and will impact every corner of society. In particular, AI / ML (machine learning) will shape how communication networks, a lifeline of our society, will be run. Many companies in the ICT sector are exploring how to make best use of AI/ML. ITU has been at the forefront of this endeavour exploring how to best apply AI/ML in future networks including 5G networks. The time is therefore right to bring together the technical community and stakeholders to brainstorm, innovate and solve relevant problems in 5G using AI/ML. Building on its standards work, ITU is conducting a global ITU AI/ML 5G Challenge on the theme “How to apply ITU's ML architecture in 5G networks". Participants will be able to solve real world problems, based on standardized technologies developed for ML in 5G networks. Teams will be required to enable, create, train and deploy ML models (such that participants will acquire hands-on experience in AI/ML in areas relevant to 5G). Participation is open to ITU Member States, Sector Members, Associates and Academic Institutions and to any individual from a country that is a member of ITU.

Participation

Theme 1 from KDDI

Analysis on Route Information Failure in IP Core Networks
by NFV-Based Test Environment

Challenge Problem

Background

The stable and high quality Internet connectivity is mandatory to 5G mobile networks, but once something unexpected happens, the influence of the defect is quite severing. In addition, the Internet is operated mutually among operators, and one failure in a domain happens to be rapidly spread all over the world. Only highly experienced operators can tackle such globally affected network failure and the anomaly detection is desired to be automatically and rapidly performed by AI/ML. Each mobile operator has at least one IP backbone network which is attached to mobile core networks. The IP backbone network interconnect with other operators’ backbone networks via border gateway routers. Border gateway routers continuously update their route information from received internal/external route information, and need to feed back and forth them appropriately. Thus, those routers play a very significant role for 5G services, and the defect in hardware/software as well as mis-operation is desired to be immediately detected to maintain a certain service level.

Problem statement

In this problem, the data sets at border gateway routers are provided for this problem along with network status information such as normal, a failure, mis-operation and so forth, as normal/abnormal labels. Participants are required to create the model to pinpoint the network status of failures and mis-operation using those data sets and evaluate the performance of the developed model.

Evaluation

Participants must create and train a model of AI/ML by the data set for learning and verify the performance of the derived model by the data set of evaluation in terms of anomaly detection and root cause analysis.

Data Set

The data sets used for this challenge were created in the NFV-based test environment simulated for a commercial IP core network according to [1-1]. In this sense, they are synthetic data, but as similar as the real data, resulting from our NFV-based test environment. The data sets consist of normal/abnormal labels, performance monitoring data sets such as traffic volume and CPU/MEM usage ratio, and route information such as Border Gateway Protocols (BGP) static metrics as well as BGP route information. Whilst the data sets were kept to be stored for a long period enough to be analysed, intentional network failures were applied to the network, leading to abnormal labels.


In order create data sets, the data collector was developed to collects and stores data sets every minute from the network. Once a failure is intentionally caused and recovered, the network indicates a failure or normal status after a period of transition, corresponding to failure data (orange arrows) and recovery data (blue arrows). The period of transition depends on a failure scenario and enough guard time is desired to be considered. The time interval between a failure and a recovery is 5 min (Fig. 1-1).




Two types of dataset files for learning and evaluation are provided to participants (Fig. 1-2). The dataset files for learning can be used for training AI models, and the dataset files for evaluation can be used for evaluating performance of the trained model. The dataset files for learning corresponds to use cases when all failure scenarios are comprehensively invoked at all possible failure points. The dataset files for evaluation corresponds to the case when a combination of a failure scenario and a failure point is randomly and limitedly generated.




[1-1] J. Kawasaki, G. Mouri, and Y. Suzuki, "Comparative Analysis of Network Fault Classification Using Machine Learning," NOMS2020, 10.1109/NOMS47738.2020.9110454.


Submission

Create and train a model of AI/ML by the data set for learning and verify the performance of the derived model by the data set of evaluation in terms of anomaly detection and root cause analysis. Participants must submit a power point file with a pdf format indicating the results and a demonstration video showing predicting performance. Please send your result to the following address by E-mail.

Subject of E-mail has to be [ITU-JP-Theme1].

Theme 2 from NEC

Network State Estimation by Analyzing Raw Video Data

Challenge Problem

Background

Due to COVID-19 pandemic, the importance of interactive live video streaming services, e.g., telework system using web cameras, has been increasing. However, the Internet cannot avoid to accommodate the increasing traffic generated from such bandwidth-consuming video streaming services, which results in heavy congestion. In case of video streaming services by over-the-top (OTT) service providers, e.g., Netflix, YouTube, and Amazon, they address the issue in COVID-19 pandemic by setting lower standard resolution based on traffic load of their services. Similarly, in case of interactive video streaming services using web cameras, video quality should be optimized based on their network state.
This situation causes a challenging issue of passive network state estimation by analyzing raw video data. Conventionally, many researchers in the field of video streaming have addressed to estimate network state by using playback buffer state. However, analyzing not KPI, e.g., bit rate and resolution, but raw video images are important for practical use cases such as telework system. Recently, we observe a new trend of artificial intelligence (AI) techniques, such as deep learning, that make a breakthrough of raw image analysis. This challenge is the first step to understand relationship between raw video images and network state.

Background of video streaming

RTP [2-1], a communication protocol suitable for live video streaming services using web cameras, is used here. Video image quality, e.g., noise, depends on the network condition (Fig. 2-1).


[2-1] H. Schulzrinne, et al., ``RTP: A Transport Protocol for Real-Time Applications," Request for Comments 3550, July 2003, https://tools.ietf.org/html/rfc3550.

Problem statement

The goal of this challenge is to estimate network state, i.e., throughput and loss ratio, from given raw video data sets. The participants are expected to train and test an AI model using the video data with labels of network state (Fig. 2-2).

Evaluation

All submission will be evaluated in terms of a performance measure (MAE) and technical excellence.

Mean absolute error ($MAE$) will be used as a measure, which is defined as follows. From the number of tests is $n$ and the $i$th estimation value is $Estimation[i]$, $MAE$ is calculated for each of bandwidth and loss ratio. \[ MAE = \frac{1}{n}\sum_{i=1}^{n} | Estimation[i] - Answer | \]

Data Set

Two types of videos are provided.
1) Original video
  We use open data as an original video. The original video follows .mp4 format (YouTube-8M).

2) Received video
  The received videos are also formatted by .mp4. In addition, file name of a video delivered in a network condition of certain bandwidth and loss ratio follows "videoid_bandwidth_loss.mp4". Datasets are generated in our lab environment (Fig. 2-3). Video Streamer (VS) transmits original video to Video Viewer (VV) via Network Emulator (NE) over RTP. Setting parameters are shown in Table 2-1.



NE control traffic rate and packet loss based on the following policy. Video traffic is shaped with predefined throughput and packets will be lost with predefined loss ratio. Dataset for training is generated on the basis of the following network condition.

    - Traffic rate: from 1100kbps to 2000kbps at 100kbps intervals
    - Packet loss ratio: 0.001%, 0.01%, 0.025%, 0.05%, 0.1%

Submission

Participants need to submit report.

-Report includes explanation of your method/approach, evaluation results (MAE, See "Evaluation criteria") for provided data set, and consideration at least.

-Report format: A4 size, pdf, 4 pages at most.

Please send your result to the following address by E-mail.

Subject of E-mail has to be [ITU-JP-Theme2].

Theme 3 from RISING

WiFi Location Estimation

Challenge Problem

This theme is not for Final Conference of ITU

Please see Japanese page. This theme is not for final conference of ITU.

  • Rules

    • If you don't have an ITU account, please create one for challenge registration according to this guide.
    • For Themes 1 and 2, it is mandatory filling the registration form at this registration form with your ITU account, if you would like to access to final conference of the ITU AI/ML in 5G Challenge.
    • If you would like to participate in Theme 3, you can not register from ITU website. Please register from Japanese Website.
  • To participate in Themes 1, 2, and 3, the following rules must be satisfied.
    • Please check the detailed information from ITU AI/ML 5G Challenge: Participation Guidelines (.docx file).
    • You can participate in teams of up to 4 members (i.e., 1-4 members).
    • This challenge is open to the two categories of participants: student and professional.
    • All the team members should be announced at the beginning (in the registration form) and will be considered to have an equal contribution.
    • The proposed solutions for Themes 1 and 2 must be fundamentally based on neural network models.
    • The proposed solution cannot use network simulation tools.
    • Solutions must be trained only with samples included in the training dataset we provide.
    • It is not allowed to use additional data obtained from other datasets or synthetically generated.
    • You may use any existing neural network architecture. However, it has to be trained from scratch and it must be clearly cited in the solution description.
    • Please send your result to the following address by E-mail.
    • During the evaluation process, each team can submit solutions many times by E-mail.
    • Subject of E-mail has to be [ITU-JP-Theme1], [ITU-JP-Theme2], or [ITU-JP-Theme3].
    • Final submission will be evaluated by organiers and committee members.

    Awards

    • Top three teams, which were registered from ITU website, for each theme have access to Final Conference of ITU AI/ML in 5G Challenge.
      • In this conference, the 3 best solutions will be awarded among all the challenges proposed under this initiative.
    • The 3 best solutions for Themes 1, 2, and 3 will be awarded by KDDI, NEC, RISING, regardless of the registration from ITU website and Japanese website.
    • Those awarded teams will be recognized in this website and certificates of appreciation will be generated for them.

Award Sponsorship

Organizers

Committee Members

  • Koichi Adachi

    Univ. of Electro-Comm.

  • Daisuke Anzai

    Nagoya Inst. of Tech.

  • Kazunori Hayashi

    Kyoto Univ.

  • Yusuke Hirota

    NICT

  • Suguru Kameda

    Tohoku Univ.

  • Ryota Kawashima

    Nagoya Inst. of Tech.

  • Tatsuaki Kimura

    Osaka Univ.

  • Kazuhiko Kinoshita

    Tokushima Univ.

  • Yoshiaki Narusue

    Univ. of Tokyo

  • Daiki Nobayashi

    Kyushu Inst. of Tech.

  • Masahiro Sasabe

    NAIST

  • Ryoichi Shinkuma

    Kyoto Univ.

  • Yusuke Shinohara

    NEC

  • Osamu Takyu

    Shinshu Univ.

  • Yuya Tarutani

    Okayama Univ.

  • Daisuke Umehara

    Kyoto Inst. of Tech.

  • Hiroshi Yamamoto

    Ritsumeikan Univ.

  • Koji Yamamoto

    Kyoto Univ.

  • Shinji Yamashita

    Fujitsu Lab.



Contact Information