Real-time compressive data collection for cloud monitoring

1. A method for compressive sensing based data collection in cloud monitoring, the method comprising:

receiving multi-dimensional data associated with performance of a cloud infrastructure collected by a plurality of probes within the cloud infrastructure;

determining a sparsity feature of the received multi-dimensional data;

applying compressive sensing to compress the multi-dimensional data into single-dimensional data using the sparsity feature, the single-dimensional data being suitable for use to reconstruct the multi-dimensional data; and

reconstructing the single-dimensional data into multi-dimensional data.

2. The method according to claim 1 , wherein each probe is a physical instance of a data source within the cloud infrastructure providing state data and the plurality of probes include one or more of a processing unit, a monitoring sensor associated with a virtual machine, and a server.

3. The method according to claim 2 , wherein the state data is associated with a property of a probe including one or more of a processor utilization, a memory utilization, a disk input/output (I/O) utilization, an adverse event associated with an application, a running time associated with an application, a resource allocation, context information, network usage, and disk temperature.

4. The method according to claim 1 , further comprising:

compressing the multi-dimensional data by separating readings of each dimension of a probe vector that includes readings from a plurality of data sources and processing the separate readings respectively.

5. The method according to claim 1 , further comprising:

compressing the multi-dimensional data by vectorizing messages from each probe and concatenating the vectorized messages into a single vector.

6. The method according to claim 1 , further comprising:

compressing the multi-dimensional data from the plurality of probes employing a sparse transformation.

7. The method according to claim 6 , wherein the sparse transformation includes one of discrete cosine transform or a wavelet transform.

8. The method according to claim 6 , further comprising:

applying compressive sampling to the transformed multi-dimensional data by multiplying a single-dimensional data vector with a random construction matrix, wherein each column of the random construction matrix includes a series of random numbers for a probe corresponding to the single-dimensional data vector.

9. A method for compressive sensing based data collection in cloud monitoring, the method comprising:

receiving a plurality of messages from a plurality of probes within a cloud infrastructure at aggregator switches of the cloud infrastructure;

generating multi-dimensional data associated with performance of the cloud infrastructure from the received messages;

determining a sparsity feature of the multi-dimensional data;

applying compressive sensing to compress the multi-dimensional data into single-dimensional messages using the sparsity feature by separating readings of each dimension of a probe vector that includes readings from a plurality of data sources and processing the separate readings respectively; and

steering the single-dimensional messages to a service manager within the cloud infrastructure to enable reconstructing of the single-dimensional messages into multi-dimensional data.

10. The method according to claim 9 , further comprising:

initializing the plurality of probes by initializing a producer thread and a consumer thread, wherein the producer thread collects data from the plurality of probes and the consumer thread reads the collected data; and

adjusting a single-dimensional message transmission rate within the cloud infrastructure according to a real-time monitoring service specification.

11. The method according to claim 9 , further comprising:

applying compressive sampling to the transformed multi-dimensional data by multiplying a single-dimensional message vector with a random construction matrix, wherein each column of the random construction matrix includes a series of random numbers for a probe corresponding to the single-dimensional message vector.

12. The method according to claim 11 , further comprising:

steering the single-dimensional messages by splitting the random construction matrix into at least three segments to create at least three single-dimensional message vectors of equal size for each probe message, steering the same-size single-dimensional message vectors to the service manager, and concatenating the single-dimensional message vectors at the service manager.

13. The method according to claim 11 , further comprising:

steering the single-dimensional messages by partitioning the random construction matrix into at least nine segments to create at least three measurement matrices, steering the measurement matrices to the service manager, and obtaining individual encoded message vectors through summation operations at top-of-rack (TOR) switches.

14. The method according to claim 13 , further comprising:

providing communication links between switches of the cloud infrastructure and global control parameters to enable the summation operations, wherein the global control parameters are generated at a data filtering module of the cloud infrastructure.

15. The method according to claim 9 , further comprising:

reconstructing the single-dimensional messages employing a linear programming optimization;

extracting sparse domain information from the reconstructed single-dimensional messages; and

transforming the reconstructed single-dimensional messages to original state domain by applying a reverse sparse domain transform.

16. A cloud-based datacenter configured to employ compressive sensing based data collection in cloud monitoring, the datacenter comprising:

a plurality of probes configured to collect data associated with performance of a plurality of nodes of a cloud infrastructure;

a plurality of aggregators configured to:

receive the collected data from the plurality of probes;

generate multi-dimensional data from the received data;

determine a sparsity feature of the multi-dimensional data; and

apply compressive sensing to compress the multi-dimensional data into single-dimensional data using the sparsity feature; and

a cloud monitoring service configured to manage steering of the single-dimensional data to a service manager, wherein the service manager is configured to reconstruct the single-dimensional data into multi-dimensional data.

17. The datacenter according to claim 16 , wherein the aggregators are configured to compress the multi-dimensional data from the plurality of probes employing a sparse transformation.

18. The datacenter according to claim 17 , wherein the sparse transformation includes one of discrete cosine transform or a wavelet transform.

19. The datacenter according to claim 16 , wherein a random global seed is broadcast from a central node of the cloud infrastructure such that each probe generates its own seed using the global seed and a probe identification to avoid transmitting the random construction matrix throughout the cloud infrastructure.

20. The datacenter according to claim 16 , wherein the cloud monitoring service is configured to:

steer the single-dimensional data directly via multiple levels of switches to the service manager within the cloud infrastructure.