Annotating datasets without redundant copying
Disclosed embodiments are methods, apparatuses, and computer-readable media for annotating distributed data without redundant data copying. In one embodiment, a method is disclosed comprising reading a raw dataset, the raw dataset comprising a first set of columns and a first set of rows; generating an annotation dataset, the annotation dataset comprising a second set of columns and a second set of rows; assigning row identifiers to each row in the second set of rows, the row identifiers aligning the second set of rows with the first set of rows based on the underlying storage of the raw dataset and annotation dataset; and writing the annotation dataset to a distributed storage medium.
1. A method comprising:
reading a raw dataset, the raw dataset comprising a first set of columns and a first set of rows;
generating an annotation dataset, the annotation dataset comprising a second set of columns and a second set of rows;
generating row identifiers based on storage characteristics of the raw dataset and the annotation dataset, the storage characteristics of the raw dataset comprising a unique identifier of the raw dataset, and one or more of a file identifier of the raw dataset or a stripe number of the raw dataset, and the storage characteristics of the annotation dataset comprising a second unique identifier of an input dataset used to generate the annotation dataset;
assigning the row identifiers to each row in the second set of rows, the row identifiers aligning the second set of rows with the first set of rows; and
writing the annotation dataset to a distributed storage medium.
2. The method of claim 1 , the reading a raw dataset comprising reading one or more physical datasets comprising a composite dataset.
3. The method of claim 1 , the generating an annotation dataset comprising mapping a file storing the annotation dataset to a file associated with the raw dataset and used to generate the annotation dataset.
4. The method of claim 1 , the generating an annotation dataset comprising mapping a file storing the annotation dataset to a stripe associated with the raw dataset and used to generate the annotation dataset.
5. The method of claim 1 , further comprising:
reading the annotation dataset;
processing one or more rows of the annotation dataset, the processing resulting in a first re-ordering of the annotation dataset;
performing a second re-ordering on the annotation dataset based on the row-identifiers; and
storing the annotation dataset as a processed annotation dataset to the distributed storage medium.
6. A non-transitory computer-readable storage medium for tangibly storing computer program instructions capable of being executed by a computer processor, the computer program instructions defining the steps of:
reading a raw dataset, the raw dataset comprising a first set of columns and a first set of rows;
generating an annotation dataset, the annotation dataset comprising a second set of columns and a second set of rows;
generating row identifiers based on storage characteristics of the raw dataset and the annotation dataset, the storage characteristics of the raw dataset comprising a unique identifier of the raw dataset, and one or more of a file identifier of the raw dataset or a stripe number of the raw dataset, and the storage characteristics of the annotation dataset comprising a second unique identifier of an input dataset used to generate the annotation dataset;
assigning the row identifiers to each row in the second set of rows, the row identifiers aligning the second set of rows with the first set of rows; and
writing the annotation dataset to a distributed storage medium.
7. The non-transitory computer-readable storage medium of claim 6 , the reading a raw dataset comprising reading one or more physical datasets comprising a composite dataset.
8. The non-transitory computer-readable storage medium of claim 6 , the generating an annotation dataset comprising mapping a file storing the annotation dataset to a file associated with the raw dataset and used to generate the annotation dataset.
9. The non-transitory computer-readable storage medium of claim 6 , the generating an annotation dataset comprising mapping a file storing the annotation dataset to a stripe associated with the raw dataset and used to generate the annotation dataset.
10. The non-transitory computer-readable storage medium of claim 6 , the instructions further defining the steps of:
reading the annotation dataset;
processing one or more rows of the annotation dataset, the processing resulting in a first re-ordering of the annotation dataset;
performing a second re-ordering on the annotation dataset based on the row-identifiers; and
storing the annotation dataset as a processed annotation dataset to the distributed storage medium.
11. An apparatus comprising:
a processor; and
a storage medium for tangibly storing thereon program logic for execution by the processor, the stored program logic causing the processor to perform the operations of:
reading a raw dataset, the raw dataset comprising a first set of columns and a first set of rows;
generating an annotation dataset, the annotation dataset comprising a second set of columns and a second set of rows;
generating row identifiers based on storage characteristics of the raw dataset and the annotation dataset, the storage characteristics of the raw dataset comprising a unique identifier of the raw dataset, and one or more of a file identifier of the raw dataset or a stripe number of the raw dataset, and the storage characteristics of the annotation dataset comprising a second unique identifier of an input dataset used to generate the annotation dataset;
assigning the row identifiers to each row in the second set of rows, the row identifiers aligning the second set of rows with the first set of rows; and
writing the annotation dataset to a distributed storage medium.
12. The apparatus of claim 11 , the generating an annotation dataset comprising mapping a file storing the annotation dataset to a file associated with the raw dataset and used to generate the annotation dataset.
13. The apparatus of claim 11 , the generating an annotation dataset comprising mapping a file storing the annotation dataset to a stripe associated with the raw dataset and used to generate the annotation dataset.
14. The apparatus of claim 11 , the stored program logic further causing the processor to perform the steps of:
reading the annotation dataset;
processing one or more rows of the annotation dataset, the processing resulting in a first re-ordering of the annotation dataset;
performing a second re-ordering on the annotation dataset based on the row-identifiers; and
storing the annotation dataset as a processed annotation dataset to the distributed storage medium.