Data storage method for massive datasets
Systems and methods for storing data in an intermediate format for storing, converting the intermediate data format into a production data format of two data volumes, and merging the two data volumes into one data volume.
1. A data processing method comprising:
storing data in an intermediate format, the intermediate format comprising at least two data volumes, and
merging and converting by a converter unit the data in an intermediate format into a production format by using cryptographic algorithm processing, the production format including an index file and a data file.
2. The data processing method of claim 1 where the index file contains a data index being expressed in a binary form and includes fixed-length data pointers to data records in its associated data file.
3. The data processing method of claim 1 where the data file comprises a plurality of data records, each data record contains a header describing at least one of a data record length and an integrity checksum.
4. The data processing method of claim 1 where the at least two data volumes are retrieved by performing a search on the index file, obtaining a data location from a pointer, and obtaining a data length by subtracting a value associated with the pointer from a selected one of the data volume length and a value associated with a pointer for a next sequential data record.
5. A data system comprising:
a storage unit that stores data in an intermediate format, comprising at least two data volumes, and
a converter unit that merges and converts the data in an intermediate format into a production format by using cryptographic algorithm processing, the production format including an index file and a data file.
6. The data system of claim 5 where the index file contains a data index being expressed in a binary form and includes fixed-length data pointers to data records in its associated data file.
7. The data system of claim 5 where the data file comprises a plurality of data records, each data record contains a header describing the data record length and an optional integrity checksum.
8. The data system of claim 5 where the at least two data volumes are retrieved by performing a search on the index file, obtaining a data location from a pointer, and obtaining a data length by subtracting a value associated with the pointer from a selected one of the data volume length and a value associated with a pointer for a next sequential data record.