Tree-like metadata structure for composite datasets
Disclosed are embodiments for generating metadata files for composite datasets. In one embodiment, a method is disclosed comprising generating a tree representing a plurality of datasets; parsing the tree into an algebraic representation of the tree; identifying a plurality of terms in the algebraic representation, each term in the terms comprising at least two factors, each of the two factors associated with a dataset in the plurality of datasets; generating a metadata object of the plurality of terms; serializing the metadata object to generate serialized terms; and storing the serialized terms in a metadata file associated with the plurality of datasets.
1. A method comprising:
generating, by a processor, a tree representing a plurality of datasets;
parsing, by the processor, the tree into an algebraic representation of the tree;
identifying, by the processor, a plurality of terms in the algebraic representation, each term in the terms comprising at least two factors, each of the two factors associated with a dataset in the plurality of datasets;
generating, by the processor, a metadata object of the plurality of terms;
serializing, by the processor, the metadata object to generate serialized terms;
storing, by the processor, the serialized terms in a metadata file associated with the plurality of datasets; and
re-constructing, by the processor, a composite dataset including the plurality of datasets in response to a command by generating at least one partial dataset based on the serialized terms.
2. The method of claim 1 , the generating a tree comprising identifying a root node of the tree, the root node comprising a right-most dataset in the plurality of datasets; and identifying one or more child nodes, the child nodes including at least one child node associated with a root dataset.
3. The method of claim 2 , the parsing the tree comprising identifying at least one vertical join and representing the vertical join as a summation.
4. The method of claim 1 , the parsing the tree comprising identifying at least one horizontal join and representing the horizontal join as a product.
5. The method of claim 1 , the identifying a plurality of terms comprising expanding the algebraic representation by factoring the algebraic representation.
6. The method of claim 1 , the serializing the metadata object comprising generating a binary representation of the plurality of terms.
7. The method of claim 1 , wherein re-constructing the composite dataset further comprises:
receiving, by the processor, the command;
loading, by the processor, the metadata file;
scanning, by the processor, the plurality of terms in the metadata file; and
building, by the processor, the composite dataset based on the scanning.
8. A non-transitory computer-readable storage medium for tangibly storing computer program instructions capable of being executed by a computer processor, the computer program instructions defining the steps of:
generating a tree representing a plurality of datasets;
parsing the tree into an algebraic representation of the tree;
identifying a plurality of terms in the algebraic representation, each term in the terms comprising at least two factors, each of the two factors associated with a dataset in the plurality of datasets;
generating a metadata object of the plurality of terms;
serializing the metadata object to generate serialized terms;
storing the serialized terms in a metadata file associated with the plurality of datasets: and
re-constructing a composite dataset including the plurality of datasets in response to a command by generating at least one partial dataset based on the serialized terms.
9. The non-transitory computer-readable storage medium of claim 8 , the generating a tree comprising identifying a root node of the tree, the root node comprising a right-most dataset in the plurality of datasets; and identifying one or more child nodes, the child nodes including at least one child node associated with a root dataset.
10. The non-transitory computer-readable storage medium of claim 8 , the parsing the tree comprising identifying at least one horizontal join and representing the horizontal join as a product.
11. The non-transitory computer-readable storage medium of claim 10 , the parsing the tree comprising identifying at least one vertical join and representing the vertical join as a summation.
12. The non-transitory computer-readable storage medium of claim 8 , the identifying a plurality of terms comprising expanding the algebraic representation by factoring the algebraic representation.
13. The non-transitory computer-readable storage medium of claim 8 , the serializing the metadata object comprising generating a binary representation of the plurality of terms.
14. The non-transitory computer-readable storage medium of claim 8 , wherein re-constructing the composite dataset further comprises:
receiving, by the processor, the command;
loading, by the processor, the metadata file;
scanning, by the processor, the plurality of terms in the metadata file; and
building, by the processor, the composite dataset based on the scanning.
15. An apparatus comprising:
a processor; and
a storage medium for tangibly storing thereon program logic for execution by the processor, the stored program logic causing the processor to perform the operations of:
generating a tree representing a plurality of datasets,
parsing the tree into an algebraic representation of the tree,
identifying a plurality of terms in the algebraic representation, each term in the terms comprising at least two factors, each of the two factors associated with a dataset in the plurality of datasets,
generating a metadata object of the plurality of terms,
serializing the metadata object to generate serialized terms,
storing the serialized terms in a metadata file associated with the plurality of datasets and
re-constructing a composite dataset including the plurality of datasets in response to a command by generating at least one partial dataset based on the serialized terms.
16. The apparatus of claim 15 , the generating a tree comprising identifying a root node of the tree, the root node comprising a right-most dataset in the plurality of datasets; and identifying one or more child nodes, the child nodes including at least one child node associated with a root dataset.
17. The apparatus of claim 15 , the parsing the tree comprising identifying at least one horizontal join and representing the horizontal join as a product.
18. The apparatus of claim 17 , the parsing the tree comprising identifying at least one vertical join and representing the vertical join as a summation.
19. The apparatus of claim 15 , the identifying a plurality of terms comprising expanding the algebraic representation by factoring the algebraic representation.
20. The apparatus of claim 15 , the serializing the metadata object comprising generating a binary representation of the plurality of terms.