Multi-objective compression for data tiering in a stream data platform
A method includes identifying, by a server module running at a stream data platform, a set of data to be moved to a target data storage tier of the stream data platform, querying, by the server module, service level agreement (SLA) objectives of the target data storage tier, fetching, by the server module, the set of data, selecting, by the server module, a compressor based on the SLA objectives, compressing the set of data using the selected compressor, and storing the set of data in the target data storage tier.
1. A method, comprising:
identifying, by a server module running at a stream data platform, a set of data to be moved to a target data storage tier of the stream data platform;
querying, by the server module, service level agreement (SLA) objectives of the target data storage tier;
fetching, by the server module, the set of data;
selecting, by the server module, a compressor based on the SLA objectives;
compressing the set of data using the selected compressor; and
storing the set of data in the target data storage tier.
2. The method as recited in claim 1 , wherein the SLA objectives specify constraints to be satisfied by a compression algorithm to be used for data to be stored in the target data storage tier.
3. The method as recited in claim 1 , wherein the set of data was compressed prior to the fetching, and the set of data is decompressed prior to being compressed with the selected compressor.
4. The method as recited in claim 1 , wherein the identified set of data is received at the stream data platform from a client.
5. The method as recited in claim 1 , wherein the set of data that is compressed using the selected compressor is in batch form and is packed together with compression metadata that identifies the selected compressor.
6. The method as recited in claim 1 , wherein the target data storage tier includes another set of data that has been compressed differently than the set of data.
7. The method as recited in claim 1 , wherein the selected compressor is a compressor that best meets the SLA objectives.
8. The method as recited in claim 1 , wherein the set of data is moved based on a change to an attribute or characteristic of the set of data.
9. The method as recited in claim 1 , wherein the set of data is moved from another tier of the stream data platform.
10. The method as recited in claim 1 , wherein, prior to the fetching, the set of data has been packed together with compression metadata in a stream batch, and the compression metadata is replaced with compression metadata identifying the selected compressor.
11. A non-transitory storage medium having stored therein instructions that are executable by one or more hardware processors to perform operations comprising:
identifying, by a server module running at a stream data platform, a set of data to be moved to a target data storage tier of the stream data platform;
querying, by the server module, service level agreement (SLA) objectives of the target data storage tier;
fetching, by the server module, the set of data;
selecting, by the server module, a compressor based on the SLA objectives;
compressing the set of data using the selected compressor; and
storing the set of data in the target data storage tier.
12. The non-transitory storage medium as recited in claim 11 , wherein the SLA objectives specify constraints to be satisfied by a compression algorithm to be used for data to be stored in the target data storage tier.
13. The non-transitory storage medium as recited in claim 11 , wherein the set of data was compressed prior to the fetching, and the set of data is decompressed prior to being compressed with the selected compressor.
14. The non-transitory storage medium as recited in claim 11 , wherein the identified set of data is received at the stream data platform from a client.
15. The non-transitory storage medium as recited in claim 11 , wherein the set of data that is compressed using the selected compressor is in batch form and is packed together with compression metadata that identifies the selected compressor.
16. The non-transitory storage medium as recited in claim 11 , wherein the target data storage tier includes another set of data that has been compressed differently than the set of data.
17. The non-transitory storage medium as recited in claim 11 , wherein the selected compressor is a compressor that best meets the SLA objectives.
18. The non-transitory storage medium as recited in claim 11 , wherein the set of data is moved based on a change to an attribute or characteristic of the set of data.
19. The non-transitory storage medium as recited in claim 11 , wherein the set of data is moved from another tier of the stream data platform.
20. The non-transitory storage medium as recited in claim 11 , wherein, prior to the fetching, the set of data has been packed together with compression metadata in a stream batch, and the compression metadata is replaced with compression metadata identifying the selected compressor.