Splitting-based approach to control data and storage growth in a computer system
In one general aspect, a method can include receiving initial content including a total number of data units for storage, identifying a number of data chunks, identifying a size for each of the data chunks, the size for each data chunk indicative of a number of data units for inclusion in the respective data chunk. The method can further include splitting the initial content into the number of data chunks, each data chunk including a portion of the total number of data units included in the initial content, the portion of the total number of data units based on the size of the data chunk, storing each of the data chunks in memory, and compressing the data chunks included in the memory.
1. A method comprising:
receiving initial content including a total number of data units for storage;
identifying a number of data chunks;
identifying a size for each of the data chunks, the size for each data chunk indicative of a number of data units for inclusion in the respective data chunk;
splitting the initial content into the number of data chunks, each data chunk including a portion of the total number of data units included in the initial content, the portion of the total number of data units based on the size of the data chunk;
storing each of the data chunks in memory in a column associated with a respective data chunk, each column including a plurality of data chunks of the identified data chunk size for the data chunk associated with the respective column; and
compressing the data chunks included in the memory.
2. The method of claim 1 ,
wherein compressing the data chunks included in the memory comprises, for each column, compressing the plurality of data chunks included in the column.
3. The method of claim 1 , wherein the memory is an in-memory database.
4. The method of claim 1 , further comprising:
creating a table view for the data chunks.
5. The method of claim 4 , further comprising:
accessing each data chunk for the initial content; and
combining each data chunk for the initial content, using the table view, to generate the initial content.
6. The method of claim 1 , wherein a sum of the number of data units included in each data chunk is equal to the total number of data units included in the initial content.
7. The method of claim 1 , wherein compressing the data chunks comprises:
identifying duplicate data chunks;
storing a single data entry for each duplicate data chunk; and
storing an additional data entry indicative of a position in the memory for each duplicate data chunk.
8. The method of claim 1 , wherein identifying a number of data chunks is based on determining that the number of data chunks results in a minimum amount of storage in the memory.
9. A computer program product, the computer program product being tangibly embodied on a non-transitory computer-readable storage medium and comprising instructions that, when executed by at least one computing device, are configured to cause the at least one computing device to:
receive initial content including a total number of data units for storage in memory;
identify a number of data chunks;
identify a size for each of the data chunks, the size for each data chunk indicative of a number of data units for inclusion in the respective data chunk;
split the initial content into the number of data chunks, each data chunk including a portion of the total number of data units included in the initial content, the portion of the total number of data units based on the size of the data chunk;
store each of the data chunks in memory in a column associated with a respective data chunk, each column including a plurality of data chunks of the identified data chunk size for the data chunk associated with the respective column; and
compress the data chunks included in the memory.
10. The computer program product of claim 9 ,
wherein the instructions configured to cause the at least one computing device to compress the data chunks included in the memory comprise instructions configured to cause the at least one computing device to, for each column, compress the plurality of data chunks included in the column.
11. The computer program product of claim 9 , wherein the memory is an in-memory database.
12. The computer program product of claim 9 , wherein the instructions are further configured to cause the at least one computing device to create a table view for the data chunks.
13. The computer program product of claim 12 , wherein the instructions are further configured to cause the at least one computing device to:
access each data chunk for the initial content; and
combine each data chunk for the initial content, using the table view, to generate the initial content.
14. The computer program product of claim 9 , wherein a sum of the number of data units included in each data chunk is equal to the total number of data units included in the initial content.
15. The computer program product of claim 9 , wherein the instructions for compressing the data chunks cause the at least one computing device to:
identify duplicate data chunks;
store a single entry for each duplicate data chunk; and
store an additional data entry indicative of a position in the memory for each duplicate data chunk.
16. A database management system comprising:
a splitting module configured to:
split initial content into a number of data chunks, each data chunk including a portion of a total number of data units included in the initial content, the portion of the total number of data units based on a size of the data chunk, the size of the data chunk indicative of a number of data units for inclusion in the data chunk;
a memory configured to:
store each of the data chunks in a column associated with a respective data chunk, each column including a plurality of data chunks of the identified data chunk size for the data chunk associated with the respective column; and
a compression module configured to:
compress the data chunks included in the memory.
17. The system of claim 16 , wherein the memory is further configured to,
for each column, compress the plurality of data chunks included in the column.
18. The system of claim 16 , further comprising a decompression module configured to create a table view for the data chunks.
19. The system of claim 18 , further comprising an assembly module configured to:
access each data chunk for the initial content; and
combine each data chunk for the initial content, using the table view, to generate the initial content.
20. The system of claim 16 ,
wherein compressing the data chunks comprises:
identifying duplicate data chunks;
storing a single data entry for each duplicate data chunk; and
storing an additional data entry indicative of a position in the memory for each duplicate data chunk.