Replacing database table join keys with index keys
Disclosed are embodiments for replacing database table join keys with index keys. In one embodiment, a method is disclosed comprising: receiving, by a processor, annotation data, the annotation data comprising a set of rows; retrieving, by the processor, a root dataset, the root dataset stored in one or more files; generating, by the processor, a row identifier for each row in the set of rows, the row identifier storing a plurality of fields enabling alignment of a respective row in the annotation data to a corresponding row in the root dataset; generating, by the processor, an annotation dataset, the annotation dataset comprising the set of rows and corresponding row identifiers; and writing, by the processor, the annotation dataset to at least one file, the at least one file separate from the one or more files.
1. A method comprising:
reading, by a processor, annotation data, the annotation data comprising a set of rows stored in a first file;
reading, by the processor, a root dataset, the root dataset stored in one or more files separate from the first file;
generating, by the processor, a row identifier for each row in the set of rows, the row identifier storing a plurality of fields representing an alignment of a respective row in the annotation data to a corresponding row in the root dataset, the row identifier comprising a concatenation of a root dataset identifier, a relative file number relative to the root dataset, and a relative row number;
generating, by the processor, an annotation dataset, the annotation dataset comprising the set of rows and corresponding row identifiers; and
writing, by the processor, the annotation dataset to at least one file, the at least one file separate from the one or more files and the first file.
2. The method of claim 1 , further comprising:
generating, by the processor, a first hash of a creation timestamp of a file selected from the one or more files;
generating, by the processor, a first hash of a file path of the file; and
concatenating, by the processor, the first hash and second hash to form the root dataset identifier.
3. The method of claim 1 , further comprising generating a value representing the file number in the root dataset corresponding to the row identifier.
4. The method of claim 1 , further comprising generating a value representing a row number in the root dataset corresponding to the row identifier.
5. The method of claim 1 , the generating the row identifier further comprising generating an offset map, the offset map defining lengths of the relative file number, relative row number, and relative row count.
6. The method of claim 1 , the generating the row identifier further comprising generating, by the processor, a relative stripe number, the relative stripe number identifying a corresponding stripe number in the root dataset.
7. The method of claim 1 , the generating the row identifier further comprising:
generating, by the processor, a relative row count; and
generating, by the processor, an input dataset identifier.
8. A non-transitory computer-readable storage medium for tangibly storing computer program instructions capable of being executed by a computer processor, the computer program instructions defining steps of:
reading, by a processor, annotation data, the annotation data comprising a set of rows stored in a first file;
reading, by the processor, a root dataset, the root dataset stored in one or more files separate from the first file;
generating, by the processor, a row identifier for each row in the set of rows, the row identifier storing a plurality of fields representing an alignment of a respective row in the annotation data to a corresponding row in the root dataset, the row identifier comprising a concatenation of a root dataset identifier, a relative file number relative to the root dataset, and a relative row number;
generating, by the processor, an annotation dataset, the annotation dataset comprising the set of rows and corresponding row identifiers; and
writing, by the processor, the annotation dataset to at least one file, the at least one file separate from the one or more files and the first file.
9. The non-transitory computer-readable storage medium of claim 8 , the steps further comprising:
generating, by the processor, a first hash of a creation timestamp of a file selected from the one or more files;
generating, by the processor, a first hash of a file path of the file; and
concatenating, by the processor, the first hash and second hash to form the root dataset identifier.
10. The non-transitory computer-readable storage medium of claim 8 , the steps further comprising generating a value representing the file number in the root dataset corresponding to the row identifier.
11. The non-transitory computer-readable storage medium of claim 8 , the steps further comprising generating a value representing a row number in the root dataset corresponding to the row identifier.
12. The non-transitory computer-readable storage medium of claim 8 , the generating the row identifier further comprising generating an offset map, the offset map defining lengths of the relative file number, relative row number, and relative row count.
13. The non-transitory computer-readable storage medium of claim 8 , the generating the row identifier further comprising generating, by the processor, a relative stripe number, the relative stripe number identifying a corresponding stripe number in the root dataset.
14. The non-transitory computer-readable storage medium of claim 8 , the generating the row identifier further comprising:
generating, by the processor, a relative row count; and
generating, by the processor, an input dataset identifier.
15. An apparatus comprising:
a processor; and
a storage medium for tangibly storing thereon program logic for execution by the processor, the stored program logic causing the processor to perform operations of:
reading annotation data, the annotation data comprising a set of rows stored in a first file;
reading a root dataset, the root dataset stored in one or more files separate from the first file;
generating a row identifier for each row in the set of rows, the row identifier storing a plurality of fields representing an alignment of a respective row in the annotation data to a corresponding row in the root dataset, the row identifier comprising a concatenation of a root dataset identifier, a relative file number relative to the root dataset, and a relative row number;
generating an annotation dataset, the annotation dataset comprising the set of rows and corresponding row identifiers; and
writing the annotation dataset to at least one file, the at least one file separate from the one or more files and the first file.
16. The apparatus of claim 15 , the operations further comprising:
generating a first hash of a creation timestamp of a file selected from the one or more files;
generating a first hash of a file path of the file; and
concatenating the first hash and second hash to form the root dataset identifier.
17. The apparatus of claim 15 , the generating the row identifier further comprising:
generating an offset map, the offset map defining lengths of the relative file number, relative row number, and relative row count; and
generating a relative stripe number, the relative stripe number identifying a corresponding stripe number in the root dataset.