Tree-like metadata structure for composite datasets
Disclosed are embodiments for generating metadata files for composite datasets. In one embodiment, a method is disclosed comprising generating a tree representing a plurality of datasets; parsing the tree into an algebraic representation of the tree; identifying a plurality of terms in the algebraic representation, each term in the terms comprising at least two factors, each of the two factors associated with a dataset in the plurality of datasets; generating a metadata object of the plurality of terms; serializing the metadata object to generate serialized terms; and storing the serialized terms in a metadata file associated with the plurality of datasets.
1. A method comprising:
loading, by a processor, a plurality of datasets from a data storage device;
converting, by the processor, a tree representing the plurality of datasets into a plurality of terms of a corresponding algebraic relationship of the tree, each term in the plurality of terms comprising at least two factors, each of the at least two factors associated with a dataset in the plurality of datasets;
serializing, by the processor, and storing the plurality of terms in a metadata file associated with the plurality of datasets in the data storage device;
receiving, by the processor, a command to re-construct a composite dataset over a network;
loading, by the processor, the metadata file from the data storage device; and
re-constructing, by the processor, the composite dataset including the plurality of datasets in response to the command by generating at least one partial dataset based on the plurality of terms read from the metadata file.
2. The method of claim 1 , further comprising generating the tree by identifying a root node of the tree, the root node comprising a right-most dataset in the plurality of datasets, and identifying one or more child nodes, the one or more child nodes including at least one child node associated with a root dataset.
3. The method of claim 2 , further comprising parsing the tree to identify a horizontal join and representing the horizontal join as a product.
4. The method of claim 3 , wherein parsing the tree comprises identifying a vertical join and representing the vertical join as a summation.
5. The method of claim 1 , further comprising identifying the plurality of terms by factoring an algebraic representation of the plurality of terms.
6. The method of claim 1 , wherein serializing the plurality of terms comprises generating a binary representation of the plurality of terms.
7. The method of claim 1 , further comprising:
receiving, by the processor, a command to access the plurality of datasets;
loading, by the processor, the metadata file;
scanning, by the processor, the plurality of terms in the metadata file; and
building, by the processor, a composite dataset based on the scanning.
8. A non-transitory computer-readable storage medium for tangibly storing computer program instructions capable of being executed by a computer processor, the computer program instructions defining steps of:
loading, by a processor, a plurality of datasets from a data storage device;
converting, by the processor, a tree representing a plurality of datasets into a plurality of terms of a corresponding algebraic relationship of the tree, each term in the plurality of terms comprising at least two factors, each of the at least two factors associated with a dataset in the plurality of datasets;
serializing, by the processor, and storing the plurality of terms in a metadata file associated with the plurality of datasets in the data storage device;
receiving, by the processor, a command to re-construct a composite dataset over a network;
loading, by the processor, the metadata file from the data storage device; and
re-constructing, by the processor, the composite dataset including the plurality of datasets in response to the command by generating at least one partial dataset based on the plurality of terms read from the metadata file.
9. The non-transitory computer-readable storage medium of claim 8 , the steps further comprising generating the tree by identifying a root node of the tree, the root node comprising a right-most dataset in the plurality of datasets, and identifying one or more child nodes, the one or more child nodes including at least one child node associated with a root dataset.
10. The non-transitory computer-readable storage medium of claim 9 , the steps further comprising parsing the tree to identify a horizontal join and representing the horizontal join as a product.
11. The non-transitory computer-readable storage medium of claim 10 , wherein parsing the tree comprises identifying a vertical join and representing the vertical join as a summation.
12. The non-transitory computer-readable storage medium of claim 8 , the steps further comprising identifying the plurality of terms by factoring an algebraic representation of the plurality of terms.
13. The non-transitory computer-readable storage medium of claim 8 , wherein serializing the plurality of terms comprises generating a binary representation of the plurality of terms.
14. The non-transitory computer-readable storage medium of claim 8 , the steps further comprising:
receiving, by the processor, a command to access the plurality of datasets;
loading, by the processor, the metadata file;
scanning, by the processor, the plurality of terms in the metadata file; and
building, by the processor, a composite dataset based on the scanning.
15. A device comprising:
a processor; and
a storage medium for tangibly storing thereon program logic for execution by the processor, the program logic including steps for:
loading, by the processor, a plurality of datasets from a data storage device;
converting a tree representing a plurality of datasets into a plurality of terms of a corresponding algebraic relationship of the tree, each term in the plurality of terms comprising at least two factors, each of the at least two factors associated with a dataset in the plurality of datasets,
serializing, by the processor, and storing the plurality of terms in a metadata file associated with the plurality of datasets in the data storage device;
receiving, by the processor, a command to re-construct a composite dataset over a network;
loading, by the processor, the metadata file from the data storage device; and
re-constructing, by the processor, the composite dataset including the plurality of datasets in response to the command by generating at least one partial dataset based on the plurality of terms read from the metadata file.
16. The device of claim 15 , the steps further comprising generating the tree by identifying a root node of the tree, the root node comprising a right-most dataset in the plurality of datasets, and identifying one or more child nodes, the one or more child nodes including at least one child node associated with a root dataset.
17. The device of claim 16 , the steps further comprising parsing the tree to identify a horizontal join and representing the horizontal join as a product.
18. The device of claim 17 , wherein parsing the tree comprises identifying a vertical join and representing the vertical join as a summation.
19. The device of claim 15 , the steps further comprising identifying the plurality of terms by factoring an algebraic representation of the plurality of terms.
20. The device of claim 15 , wherein serializing the plurality of terms comprises generating a binary representation of the plurality of terms.