Systems and methods of data auto-tiering using relativized discrepancy
Methods and systems for data auto-tiering are disclosed. According to some embodiments, the method receives a multiplicity of data streams. For each data stream, the method detects a data change within the data stream. The method further determines a magnitude of the data change. The method further assigns a tier level to the data stream based on the magnitude of the data change.
1. A computer-implemented method of data auto-tiering, comprising:
receiving a plurality of data streams; and
for each data stream,
detecting a data change within the data stream,
determining a magnitude of the data change, and
assigning a tier level to the data stream based on the magnitude of the data change.
2. The method of claim 1 , wherein detecting the data change within the data stream comprises:
using a first sliding window on a data point where a last data change was detected, and
using a second sliding window that slides a step forward every time there is new data in the data stream.
3. The method of claim 2 , wherein determining the magnitude of the data change comprises: evaluating a supremum based on a first probabilistic distribution associated with the first sliding window and a second probabilistic distribution associated with the second sliding window.
4. The method of claim 1 , wherein the assigned tier level is a lowest tier level representing minimum bandwidth and storage, a medium tier level representing optimum bandwidth and storage, or a highest tier level representing maximum bandwidth and storage.
5. The method of claim 1 , wherein the magnitude of the data change is a factor of a predetermined value.
6. The method of claim 5 , wherein the factor is between 0 and 0.5, greater than 0.5 but less than 0.75, or greater than 0.75.
7. The method of claim 6 , wherein the predetermined value is 1.
8. The method of claim 1 , wherein the detection of the data change, the determination of the magnitude of the data change, and the assignment of the tier level to the data stream are run in parallel for each of the data streams.
9. A non-transitory machine-readable medium having instructions stored therein, which when executed by a processor, cause the processor to perform operations, the operations comprising:
receiving a plurality of data streams; and
for each data stream,
detecting a data change within the data stream,
determining a magnitude of the data change, and
assigning a tier level to the data stream based on the magnitude of the data change.
10. The non-transitory machine-readable medium of claim 9 , wherein detecting the data change within the data stream comprises:
using a first sliding window on a data point where a last data change was detected, and
using a second sliding window that slides a step forward every time there is new data in the data stream.
11. The non-transitory machine-readable medium of claim 10 , wherein determining the magnitude of the data change comprises: evaluating a supremum based on a first probabilistic distribution associated with the first sliding window and a second probabilistic distribution associated with the second sliding window.
12. The non-transitory machine-readable medium of claim 9 , wherein the assigned tier level is a lowest tier level representing minimum bandwidth and storage, a medium tier level representing optimum bandwidth and storage, or a highest tier level representing maximum bandwidth and storage.
13. The non-transitory machine-readable medium of claim 9 , wherein the magnitude of the data change is a factor of a predetermined value.
14. The non-transitory machine-readable medium of claim 13 , wherein the factor is between 0 and 0.5, greater than 0.5 but less than 0.75, or greater than 0.75.
15. The non-transitory machine-readable medium of claim 14 , wherein the predetermined value is 1.
16. The non-transitory machine-readable medium of claim 9 , wherein the detection of the data change, the determination of the magnitude of the data change, and the assignment of the tier level to the data stream are run in parallel for each of the data streams.
17. A data processing system, comprising:
a processor; and
a memory coupled to the processor to store instructions, which when executed by the processor, cause the processor to perform operations, the operations comprising:
receiving a plurality of data streams; and
for each data stream,
detecting a data change within the data stream,
determining a magnitude of the data change, and
assigning a tier level to the data stream based on the magnitude of the data change.
18. The data processing system of claim 17 , wherein detecting the data change within the data stream comprises:
using a first sliding window on a data point where a last data change was detected, and
using a second sliding window that slides a step forward every time there is new data in the data stream.
19. The data processing system of claim 18 , wherein determining the magnitude of the data change comprises: evaluating a supremum based on a first probabilistic distribution associated with the first sliding window and a second probabilistic distribution associated with the second sliding window.
20. The data processing system of claim 17 , wherein the assigned tier level is a lowest tier level representing minimum bandwidth and storage, a medium tier level representing optimum bandwidth and storage, or a highest tier level representing maximum bandwidth and storage.