Data persistence formats management for column loadable columns

1 . A system for managing data persistence formats in a computing system, comprising:

at least one data processor; and

at least one memory result in operations comprising:

computing an estimated serial size(ES) for a dataset by assessing a cumulative size of data components within the dataset;

computing an estimated unified size (EU) for the dataset by assessing an individual size of each of the data components separately;

selecting, based at least in part on a predefined set of rules, between a serial persistence format and a unified persistence format to store the dataset, wherein the selection is based at least in part on the computed estimated serial size and the computed estimated unified size;

storing the selected persistence format in a metadata associated with the stored dataset; and

in response to a loading request, loading the stored dataset by referring to the metadata,

wherein the predefined set of rules comprises:

a determination of whether the ES is less than a minimum size threshold for unified storage, and if so, selecting the serial persistence format;

a determination of whether the ES is greater than a maximum size threshold for serial storage, and if so, selecting the unified persistence format;

a comparison of the ES and the EU, and selecting the unified persistence format if the ES is greater than the EU; and

a calculation of a percentage difference between the ES and the EU, and selecting the unified persistence format if the percentage difference is within a predetermined threshold.

2 . The system of claim 1 , wherein a page-loadable format for loading the stored data set is available if the selected persistence format for the stored dataset is the unified persistence format, and the page-loadable format for loading the dataset is not available if the selected persistence format for the dataset is the serial persistence format.

3 . The system of claim 1 , wherein the operations further comprises:

detecting a scenario where the minimum size threshold for unified storage is greater than the maximum size threshold for serial storage; and

generating a reconfiguration request for dynamic format selection.

4 . The system of claim 1 , wherein the dataset comprises columns and components of columns, and the selection of the persistence format is determined independently for each component.

5 . The system of claim 4 , wherein the operations further comprise:

determining if a dictionary component of the dataset has a size that is below a dictionary sized threshold, and

storing the dictionary component to serial persistence format in response to the size below the dictionary size threshold.

6 . The system of claim 1 , wherein the predefined set of rules comprises identifying small columns based on predetermined thresholds for column sizes.

7 . A computer-implemented method, comprising:

computing an estimated serial size(ES) for a dataset by assessing a cumulative size of data components within the dataset;

computing an estimated unified size (EU) for the dataset by assessing an individual size of each of the data components separately;

selecting, based at least in part on a predefined set of rules, between a serial persistence format and a unified persistence format to store the dataset, wherein the selection is based at least in part on the computed estimated serial size and the computed estimated unified size;

storing the selected persistence format in a metadata associated with the stored dataset; and

in response to a loading request, loading the stored dataset by referring to the metadata,

wherein the predefined set of rules comprises:

a determination of whether the ES is less than a minimum size threshold for unified storage, and if so, selecting the serial persistence format;

a determination of whether the ES is greater than a maximum size threshold for serial storage, and if so, selecting the unified persistence format;

a comparison of the ES and the EU, and selecting the unified persistence format if the ES is greater than the EU; and

a calculation of a percentage difference between the ES and the BU, and selecting the unified persistence format if the percentage difference is within a predetermined threshold.

8 . The method of claim 7 , wherein a page-loadable format for loading the stored data set is available if the selected persistence format for the stored dataset is the unified persistence format, and the page-loadable format for loading the dataset is not available if the selected persistence format for the dataset is the serial persistence format.

9 . The method of claim 7 , wherein the operations further comprises:

detecting a scenario where the minimum size threshold for unified storage is greater than the maximum size threshold for serial storage; and

generating a reconfiguration request for dynamic format selection.

10 . The method of claim 7 , wherein the dataset comprises columns and components of columns, and the selection of the persistence format is determined independently for each component.

11 . The method of claim 10 , further comprising:

determining if a dictionary component of the dataset has a size that is below a dictionary sized threshold, and

storing the dictionary component to serial persistence format in response to the size below the dictionary size threshold.

12 . The method of claim 7 , wherein the predefined set of rules comprises identifying small columns based on predetermined thresholds for column sizes.

13 . A non-transitory computer-readable medium storing instructions, which when executed by at least one data processor, result in operations comprising:

computing an estimated serial size(ES) for a dataset by assessing a cumulative size of data components within the dataset;

computing an estimated unified size (EU) for the dataset by assessing an individual size of each of the data components separately;

selecting, based at least in part on a predefined set of rules, between a serial persistence format and a unified persistence format to store the dataset, wherein the selection is based at least in part on the computed estimated serial size and the computed estimated unified size;

storing the selected persistence format in a metadata associated with the stored dataset; and

in response to a loading request, loading the stored dataset by referring to the metadata,

wherein the predefined set of rules comprises:

a determination of whether the ES is less than a minimum size threshold for unified storage, and if so, selecting the serial persistence format;

a determination of whether the ES is greater than a maximum size threshold for serial storage, and if so, selecting the unified persistence format;

a comparison of the ES and the EU, and selecting the unified persistence format if the BS is greater than the BU; and

a calculation of a percentage difference between the ES and the EU, and selecting the unified persistence format if the percentage difference is within a predetermined threshold.

14 . The non-transitory computer-readable medium of claim 13 , wherein a page-loadable format for loading the stored data set is available if the selected persistence format for the stored dataset is the unified persistence format, and the page-loadable format for loading the dataset is not available if the selected persistence format for the dataset is the serial persistence format.

15 . The non-transitory computer-readable medium of claim 13 , wherein the operations further comprises:

detecting a scenario where the minimum size threshold for unified storage is greater than the maximum size threshold for serial storage; and

generating a reconfiguration request for dynamic format selection.

16 . The non-transitory computer-readable medium of claim 13 , wherein the dataset comprises columns and components of columns, and the selection of the persistence format is determined independently for each component.

17 . The non-transitory computer-readable medium of claim 13 , wherein the predefined set of rules comprises identifying small columns based on predetermined thresholds for column sizes.