Large Scale Monitoring and Online Analysis in a Distributed Virtualized Environment

R. Mehrotra, Abhishek Dubey, S. Abdelwahed, W. Monceaux

2011 Eighth IEEE International Conference and Workshops on Engineering of Autonomic and Autonomous Systems 2011

PDF DOI

Why This Matters

Virtualization has created new challenges for performance monitoring since multiple applications share physical resources and dynamic migration changes system configuration. This work innovates by combining multiple monitoring utilities with advanced system modeling techniques including exponential Kalman filters to predict computational resource requirements and power consumption. The integration of online monitoring with system modeling enables dynamic control of virtualized environments to maintain QoS requirements while minimizing operational costs.

What We Did

This paper describes a large-scale event-based monitoring approach for distributed systems in virtualized environments using comprehensive measurement of system variables including CPU, memory, disk utilization, and application-level metrics. The framework measures both physical/virtual CPU utilization and application variables like queue waiting time and service time. It provides extensive data processing utilities including synchronization scripts, monitoring sensors, Xenmon integration, power consumption modeling, and Kalman filter-based system identification techniques.

Key Results

Experiments in virtualized environments demonstrate accurate prediction of system behavior and effective online analysis of monitoring data. The framework successfully tracks system state changes across multiple virtual machines with minimal latency, and the Kalman filter approach enables accurate estimation of service time and delay without additional performance burden. Integration with a feedback controller shows effective management of system resources to maintain predefined QoS parameters.

Full Abstract

Cite This Paper

@inproceedings{Mehrotra2011,
  author = {Mehrotra}, R. and Dubey, Abhishek and {Abdelwahed}, S. and {Monceaux}, W.},
  booktitle = {2011 Eighth IEEE International Conference and Workshops on Engineering of Autonomic and Autonomous Systems},
  title = {Large Scale Monitoring and Online Analysis in a Distributed Virtualized Environment},
  year = {2011},
  month = {apr},
  pages = {1-9},
  abstract = {Due to increase in number and complexity of the large scale systems, performance monitoring and multidimensional quality of service (QoS) management has become a difficult and error prone task for system administrators. Recently, the trend has been to use virtualization technology, which facilitates hosting of multiple distributed systems with minimum infrastructure cost via sharing of computational and memory resources among multiple instances, and allows dynamic creation of even bigger clusters. An effective monitoring technique should not only be fine grained with respect to the measured variables, but also should be able to provide a high level overview of the distributed systems to the administrator of all variables that can affect the QoS requirements. At the same time, the technique should not add performance burden to the system. Finally, it should be integrated with a control methodology that manages performance of the enterprise system. In this paper, a systematic distributed event based (DEB) performance monitoring approach is presented for distributed systems by measuring system variables (physical/virtual CPU utilization and memory utilization), application variables (application queue size, queue waiting time, and service time), and performance variables (response time, throughput, and power consumption) accurately with minimum latency at a specified rate. Furthermore, we have shown that proposed monitoring approach can be utilized to provide input to an application monitoring utility to understand the underlying performance model of the system for a successful on-line control of the distributed systems for achieving predefined QoS parameters.},
  category = {conference},
  contribution = {colab},
  doi = {10.1109/EASe.2011.17},
  file = {:Mehrotra2011-Large_Scale_Monitoring_and_Online_Analysis_in_a_Distributed_Virtualized_Environment.pdf:PDF},
  issn = {2168-1872},
  keywords = {virtualized systems, quality of service, system modeling, power management, Kalman filtering, event-based monitoring},
  tag = {platform},
  month_numeric = {4}
}
Quick Info
Year 2011
DOI 10.1109/EASe.2011.17
Keywords
virtualized systems quality of service system modeling power management Kalman filtering event-based monitoring
Research Areas
middleware CPS ML for CPS
Search Tags

Large, Scale, Monitoring, Online, Analysis, Distributed, Virtualized, Environment, virtualized systems, quality of service, system modeling, power management, Kalman filtering, event-based monitoring, middleware, CPS, ML for CPS, 2011, Mehrotra, Dubey, Abdelwahed, Monceaux