| Presentation | 2021-08-27 An anomaly detection method to reduce the effect of concept drift Jaiswal Satish Kumar, Masuda Mineyoshi, |
|---|---|
| PDF Download Page |  PDF download Page Link |
| Abstract(in Japanese) | (See Japanese page) |
| Abstract(in English) | There is an increasing trend to use machine learning models for monitoring anomalous behavior of CPU, memory, network, and disk. However, they fail to detect anomalies immediately after abrupt concept drifts unless re-trained withpost drift data. It may take from days to weeks to collect post drift data, interrupting accurate monitoring. We identify thatit is possible to avoid re-training if the concept drifts were caused by events such as change in number of CPU, memorysize, unscheduled deletion of log files, transfer of files, etc. Since these events are expected to become more frequent dueto adoption of container based microservice architecture, it is important to enable accurate anomaly detection immediatelyafter these events. Our proposed method includes two major steps. First, we confirm that an abrupt concept drift is due toabove-mentioned events. Second, we find a transformation function which can undo the change in data distribution caused bythese events. This allows us to use the same model even after the concept drift. We evaluated our method against well-knownadaptive methods such as Adaptive Random Forest. We found that Adaptive Random Forest takes 2 weeks to adapt while theproposed method can adapt immediately. |
| Keyword(in Japanese) | (See Japanese page) |
| Keyword(in English) | AIOpsAnomaly DetectionConcept driftConcept drift adaptationLinear transformation |
| Paper # | SWIM2021-20,SC2021-18 |
| Date of Issue | 2021-08-20 (SWIM, SC) |
| Conference Information | |
| Committee | SWIM / SC |
|---|---|
| Conference Date | 2021/8/27(1days) |
| Place (in Japanese) | (See Japanese page) |
| Place (in English) | Online |
| Topics (in Japanese) | (See Japanese page) |
| Topics (in English) | |
| Chair | Kenji Saotome(Hosei Univ.) / Shinji Kikuchi(NIMS) |
| Vice Chair | Akihiro Hayashi(Shizuoka Inst. of Science and Tech.) / Yoji Yamato(NTT) / Kosaku Kimura(Fujitsu Lab.) |
| Secretary | Akihiro Hayashi(Tokyo Univ. of Science) / Yoji Yamato(Osaka Sangyo Univ.) / Kosaku Kimura(Kobe Univ.) |
| Assistant | Tsukasa Kudo(Shizuoka Inst. of Science and Tech.) / Kokichi Tsuji(Aichi Pref. Univ.) / Shin Tezuka(Hitachi) / Takao Nakaguchi(KCGI) |
| Paper Information | |
| Registration To | Technical Committee on Software Interprise Modeling / Technical Committee on Service Computing |
|---|---|
| Language | ENG-JTITLE |
| Title (in Japanese) | (See Japanese page) |
| Sub Title (in Japanese) | (See Japanese page) |
| Title (in English) | An anomaly detection method to reduce the effect of concept drift |
| Sub Title (in English) | |
| Keyword(1) | AIOpsAnomaly DetectionConcept driftConcept drift adaptationLinear transformation |
| 1st Author's Name | Jaiswal Satish Kumar |
| 1st Author's Affiliation | Hitachi, Ltd. Research & Development Group(Hitachi) |
| 2nd Author's Name | Masuda Mineyoshi |
| 2nd Author's Affiliation | Hitachi, Ltd. Research & Development Group(Hitachi) |
| Date | 2021-08-27 |
| Paper # | SWIM2021-20,SC2021-18 |
| Volume (vol) | vol.121 |
| Number (no) | SWIM-156,SC-157 |
| Page | pp.pp.46-51(SWIM), pp.46-51(SC), |
| #Pages | 6 |
| Date of Issue | 2021-08-20 (SWIM, SC) |