Sorting unsorted rows of a composite dataset after a join operation
Disclosed are embodiments for sorting rows of a dataset after a JOIN operation. In one embodiment, a method is disclosed comprising performing a JOIN operation on an annotation dataset, the performing of the JOIN operation generating an unordered dataset; grouping a plurality of rows in the unordered dataset into a plurality of buckets, the grouping performed based on a root dataset associated with the annotation dataset; sorting each bucket, the sorting comprising sorting each bucket independently; and combining each sorted bucket into a sorted dataset.
1. A method comprising:
storing a root dataset, the root dataset storing a plurality of rows;
storing an annotation dataset managed independently from the root dataset, the annotation dataset comprising a plurality of fields, each field in the plurality of fields comprising annotation data and a row identifier, the row identifier comprising a data structure storing a positional mapping to the root dataset and a row in the plurality of rows;
performing a JOIN operation on the annotation dataset and the root dataset, the performing of the JOIN operation generating an unordered dataset, wherein a first set of splits in the root dataset in the unordered dataset are misaligned relative to a corresponding second set of splits of the annotation dataset, wherein the first set of splits and the second set of splits are misaligned relative to split identifiers present in both the first set of splits and the second set of splits;
grouping a plurality of rows in the unordered dataset into a plurality of buckets, the grouping performed based on the first set of splits, wherein at least one bucket in the plurality of buckets includes a first split in the first set of splits and a second split in the second set of splits, the first split and second splits comprising misaligned splits;
sorting each bucket, the sorting comprising sorting each bucket independently; and
combining each sorted bucket into a sorted dataset.
2. The method of claim 1 , the performing a JOIN operation comprising joining the annotation dataset to the root dataset.
3. The method of claim 1 , further comprising generating a mapping of files and splits of the annotation dataset to corresponding files and splits of the root dataset.
4. The method of claim 3 , the generating the mapping comprising reversing an annotation-to-root mapping stored within a metadata file associated with the annotation dataset.
5. The method of claim 3 , the grouping the plurality of rows comprising:
extracting a relative split number for each of the plurality of rows; and
mapping each of the relative split numbers to a split number of the root dataset.
6. The method of claim 3 , the grouping the plurality of rows further comprising extracting a relative row number and relative split number associated with the row.
7. The method of claim 6 , the sorting comprising sorting the rows using only the relative row number and split number.
8. A non-transitory computer readable storage medium for tangibly storing computer program instructions capable of being executed by a computer processor, the computer program instructions defining steps of:
storing a root dataset, the root dataset storing a plurality of rows;
storing an annotation dataset managed independently from the root dataset, the annotation dataset comprising a plurality of fields, each field in the plurality of fields comprising annotation data and a row identifier, the row identifier comprising a data structure storing a positional mapping to the root dataset and a row in the plurality of rows;
performing a JOIN operation on the annotation dataset and the root dataset, the performing of the JOIN operation generating an unordered dataset, wherein a first set of splits in the root dataset in the unordered dataset are misaligned relative to a corresponding second set of splits of the annotation dataset, wherein the first set of splits and the second set of splits are misaligned relative to split identifiers present in both the first set of splits and the second set of splits;
grouping a plurality of rows in the unordered dataset into a plurality of buckets, the grouping performed based on the first set of splits;
sorting each bucket, the sorting comprising sorting each bucket independently; and
combining each sorted bucket into a sorted dataset.
9. The non-transitory computer readable storage medium of claim 8 , the performing a JOIN operation comprising joining the annotation dataset to the root dataset.
10. The non-transitory computer readable storage medium of claim 8 , the computer program instructions further defining the step of generating a mapping of files and splits of the annotation dataset to corresponding files and splits of the root dataset.
11. The non-transitory computer readable storage medium of claim 10 , the generating the mapping comprising reversing an annotation-to-root mapping stored within a metadata file associated with the annotation dataset.
12. The non-transitory computer readable storage medium of claim 10 , the grouping the plurality of rows comprising:
extracting a relative split number for each of the plurality of rows; and
mapping each of the relative split numbers to a split number of the root dataset.
13. The non-transitory computer readable storage medium of claim 10 , the grouping the plurality of rows further comprising extracting a relative row number and relative split number associated with the row.
14. The non-transitory computer readable storage medium of claim 13 , the sorting comprising sorting the rows using only the relative row number and split number.
15. An apparatus comprising:
a processor; and
a storage medium for tangibly storing thereon program logic for execution by the processor, the stored program logic causing the processor to perform operations of:
storing a root dataset, the root dataset storing a plurality of rows;
storing an annotation dataset managed independently from the root dataset, the annotation dataset comprising a plurality of fields, each field in the plurality of fields comprising annotation data and a row identifier, the row identifier comprising a data structure storing a positional mapping to the root dataset and a row in the plurality of rows;
performing a JOIN operation on the annotation dataset and the root dataset, the performing of the JOIN operation generating an unordered dataset, wherein a first set of splits in the root dataset in the unordered dataset are misaligned relative to a corresponding second set of splits of the annotation dataset, wherein the first set of splits and the second set of splits are misaligned relative to split identifiers present in both the first set of splits and the second set of splits;
grouping a plurality of rows in the unordered dataset into a plurality of buckets, the grouping performed based on the first set of splits;
sorting each bucket, the sorting comprising sorting each bucket independently; and
combining each sorted bucket into a sorted dataset.
16. The apparatus of claim 15 , the performing a JOIN operation comprising joining the annotation dataset to the root dataset.
17. The apparatus of claim 15 , further comprising generating a mapping of files and splits of the annotation dataset to corresponding files and splits of the root dataset.
18. The apparatus of claim 17 , the generating the mapping comprising reversing an annotation-to-root mapping stored within a metadata file associated with the annotation dataset.
19. The apparatus of claim 17 , the grouping the plurality of rows comprising:
extracting a relative split number for each of the plurality of rows; and
mapping each of the relative split numbers to a split number of the root dataset.
20. The apparatus of claim 17 , the grouping the plurality of rows further comprising extracting a relative row number and relative split number associated with the row; and sorting the rows using only the relative row number and split number.