Systems and methods for managing storage system monitoring data using predictive compression
A monitoring system can generate compressed storage system monitoring data segments using monitoring data obtained from a storage system. The monitoring system can obtain compression information for a compressed segment generated from a segment of storage system monitoring data and generate a predicted portion of storage system monitoring data using the compression information. The monitoring system can obtain an additional portion of storage system monitoring data, the additional portion contiguous to the segment of storage system monitoring data, and determine that the predicted portion matches the additional portion and combining the additional portion and the compressed segment. In response to a user query, the monitoring system can perform at least one of: reconstructing and providing the additional portion using the compressed segment; or providing the compressed segment for reconstruction of the additional portion.
1 . A storage system monitoring method, comprising:
obtaining compression information associated with a compressed segment generated from a segment of storage system monitoring data;
generating a predicted portion of storage system monitoring data using the compression information;
obtaining an additional portion of storage system monitoring data, the additional portion of the storage system monitoring data contiguous to the segment of storage system monitoring data;
determining the predicted portion of the storage system monitoring data matches the additional portion of the storage system monitoring data and combining the additional portion of the storage system monitoring data and the compressed segment generated from the segment of storage system monitoring data; and
in response to a query, performing at least one of:
reconstructing and providing the additional portion of the storage system monitoring data using the compressed segment generated from the segment of storage system monitoring data; or
providing the compressed segment generated from the segment of storage system monitoring data for reconstruction of the additional portion of the storage system monitoring data.
2 . The storage system monitoring method of claim 1 , wherein:
the additional portion of storage system monitoring data includes a batch of observations.
3 . The storage system monitoring method of claim 1 , wherein:
the generation of the compressed segment preprocessing an obtained original portion of storage system data; and
the method further includes, prior to determining the predicted portion matches the additional portion of the storage system monitoring data, preprocessing the additional portion of storage system monitoring data.
4 . The storage system monitoring method of claim 1 , wherein:
generating the predicted portion of storage system monitoring data using the compression information comprises:
reconstructing the compressed segment using the compression information; and
generating the predicted portion based on the reconstruction of the compressed
segment generated from the segment of storage system monitoring data.
5 . The storage system monitoring method of claim 1 , wherein:
the compression information specifies a parameterized formula that relates compressed segment indices to compressed segment values; and
the predicted portion of storage system monitoring data is generated using the parameterized formula.
6 . The storage system monitoring method of claim 1 , wherein:
determining the predicted portion matches the additional portion of the storage system monitoring data comprises determining a function of a difference between the predicted portion of the storage system monitoring data and the additional portion of the storage system monitoring data.
7 . The storage system monitoring method of claim 6 , wherein:
the function comprises a weighted or unweighted L2 norm, L1 norm, L-infinity norm, mean squared error, or Manhattan distance.
8 . The storage system monitoring method of claim 1 , wherein:
combining the additional portion of the storage system monitoring data and the compressed segment generated from the segment of storage system monitoring data comprises updating time information for the compressed segment generated from the segment of storage system monitoring data to reference the additional portion of the storage system monitoring data.
9 . The storage system monitoring method of claim 1 , wherein:
the storage system monitoring data includes CPU I/O wait time, CPU Guest Usage, CPU usage, System Status, number of connected clients, network usage, memory usage, disk usage, read latency, write latency, or operating system load.
10 . The storage system monitoring method of claim 1 , wherein:
the compression information generated from the segment of storage system monitoring data indicates that the segment of storage system monitoring data was compressed using polynomial approximation, linear predictive coding, or a frequency domain compression technique.
11 . A monitoring system, comprising:
at least one processor; and
at least one non-transitory computer readable medium containing instructions that, when executed by the at least one processor, cause the monitoring system to perform operations comprising:
obtaining compression information associated with a compressed segment generated from a segment of storage system monitoring data;
generating a predicted portion of storage system monitoring data using the compression information;
obtaining an additional portion of storage system monitoring data, the additional portion of the storage system monitoring data contiguous to the segment of storage system monitoring data;
determining the predicted portion of the storage system monitoring data matches the additional portion of the storage system monitoring data and combining the additional portion of the storage system monitoring data and the compressed segment generated from the segment of storage system monitoring data; and
in response to a query, performing at least one of:
reconstructing and providing the additional portion of the storage system monitoring data using the compressed segment generated from the segment of storage system monitoring data; or
providing the compressed segment generated from the segment of storage system monitoring data for reconstruction of the additional portion of the storage system monitoring data.
12 . The system of claim 11 , wherein:
the generation of the compressed segment includes preprocessing an obtained original portion of storage system data; and
the operations further include, prior to determining the predicted portion matches the additional portion of the storage system monitoring data, preprocessing the additional portion of storage system monitoring data.
13 . The system of claim 11 , wherein:
generating the predicted portion of storage system monitoring data using the compression information comprises:
reconstructing the compressed segment generated from the segment of storage system monitoring data using the compression information; and
generating the predicted portion of the storage system monitoring data based on the reconstruction of the compressed segment generated from the segment of storage system monitoring data.
14 . The system of claim 11 , wherein:
the compression information specifies a parameterized formula that relates compressed segment indices to compressed segment values; and
the predicted portion of storage system monitoring data is generated using the parameterized formula.
15 . The system of claim 11 , wherein:
determining the predicted portion generated from the segment of storage system monitoring data matches the additional portion generated from the segment of storage system monitoring data comprises determining a function of a difference between the predicted portion generated from the segment of storage system monitoring data and the additional portion generated from the segment of storage system monitoring data, the function comprising a weighted or unweighted L2 norm, L1 norm, L-infinity norm, mean squared error, or Manhattan distance.
16 . The system of claim 11 , wherein:
combining the additional portion of the storage system monitoring data and the compressed segment generated from the segment of storage system monitoring data comprises updating time information for the compressed segment generated from the segment of storage system monitoring data to reference the additional portion of the storage system monitoring data.
17 . The system of claim 11 , wherein:
the storage system monitoring data includes a table metric channel, a message metric channel, a streaming metric channel, a compaction metric channel, a commit log metric channel, a storage metric channel, a hint metric channel, an index metric channel, a buffer pool metric channel, a client management metric channel, a batch metric channel, or a virtual machine metric channel.
18 . A non-transitory, computer-readable medium containing instructions that, when executed by at least one processor of a system, cause the system to perform operations comprising:
obtaining compression information associated with a compressed segment generated from a segment of storage system monitoring data;
generating a predicted portion of storage system monitoring data using the compression information;
obtaining an additional portion of storage system monitoring data, the additional portion of the storage system monitoring data including a batch of observations and being contiguous to the segment of storage system monitoring data;
determining the predicted portion of the storage system monitoring data matches the additional portion of the storage system monitoring data and combining the additional portion of the storage system monitoring data and the compressed segment generated from the segment of storage system monitoring data; and
in response to a query, performing at least one of:
reconstructing and providing the additional portion of the storage system monitoring data using the compressed segment generated from the segment of storage system monitoring data; or
providing the compressed segment generated from the segment of storage system monitoring data for reconstruction of the additional portion of the storage system monitoring data.
19 . The non-transitory, computer-readable medium of claim 18 , wherein:
the segment of storage system monitoring data is compressed using polynomial approximation, linear predictive coding, or frequency domain compression, the compression information specifies a parameterized formula that relates compressed segment indices to compressed segment values; and
the predicted portion of storage system monitoring data is generated using the parameterized formula.
20 . The non-transitory, computer-readable medium of claim 18 , wherein:
determining the predicted portion of the storage system monitoring data matches the additional portion of the storage system monitoring data comprises determining a function of a difference between the predicted portion of the storage system monitoring data and the additional portion of the storage system monitoring data, the function comprising a weighted or unweighted L2 norm, L1 norm, L-infinity norm, mean squared error, or Manhattan distance.
21 . The non-transitory, computer-readable medium of claim 18 , wherein:
combining the additional portion of the storage system monitoring data and the compressed segment comprises updating time information for the compressed segment generated from the segment of storage system monitoring data to reference the additional portion.