Annotating datasets without redundant copying
Disclosed embodiments are methods, apparatuses, and computer-readable media for annotating distributed data without redundant data copying. In one embodiment, a method is disclosed comprising reading a raw dataset, the raw dataset comprising a first set of columns and a first set of rows; generating an annotation dataset, the annotation dataset comprising a second set of columns and a second set of rows; assigning row identifiers to each row in the second set of rows, the row identifiers aligning the second set of rows with the first set of rows based on the underlying storage of the raw dataset and annotation dataset; and writing the annotation dataset to a distributed storage medium.
1 . A method comprising:
reading, by a device, a dataset comprising data stored in an unordered manner in a first set of columns and a first set of rows in a database;
determining, by the device, an attribute for the data of the dataset;
partitioning, by the device, the data based on the determined attribute;
reordering, by the device, the partitioned data based on boundaries of each partition of the data, the reordering performed via a stripe-based alignment of the partitioned data that results in a reduction of a quantity of data files associated with the dataset;
modifying, by the device, the dataset by flattening the reordered data; and
storing, by the device, the modified dataset in the database.
2 . The method of claim 1 , wherein the reordering comprises modifying a configuration of the data within the dataset respective to the first set of columns and first set of rows.
3 . The method of claim 1 , wherein the reordering comprises grouping the data within the dataset based at least on the determined attribute.
4 . The method of claim 1 , further comprising:
sorting, based on attributes of the data, the data within the dataset, the sorting being contingent upon the reordering of the data within the database based on the determined attribute.
5 . The method of claim 4 , wherein the sorting is performed in relation to the first set of rows.
6 . The method of claim 4 , wherein the flattening of the reordered data is based on the sorted data.
7 . The method of claim 1 , further comprising:
formatting the flattened reordered data into a first format, the formatting enabling the storage of the modified dataset.
8 . A non-transitory computer-readable storage medium tangibly encoded with computer-executable instructions, that when executed by a device, perform a method comprising:
reading, by a device, a dataset comprising data stored in an unordered manner in a first set of columns and a first set of rows in a database;
determining, by the device, an attribute for the data of the dataset;
partitioning, by the device, the data based on the determined attribute;
reordering, by the device, the partitioned data based on boundaries of each partition of the data, the reordering performed via a stripe-based alignment of the partitioned data that results in a reduction of a quantity of data files associated with the dataset;
modifying, by the device, the dataset by flattening the reordered data; and
storing, by the device, the modified dataset in the database.
9 . The non-transitory computer-readable storage medium of claim 8 , wherein the reordering comprises modifying a configuration of the data within the dataset respective to the first set of columns and first set of rows.
10 . The non-transitory computer-readable storage medium of claim 8 , wherein the reordering comprises grouping the data within the dataset based at least on the determined attribute.
11 . The non-transitory computer-readable storage medium of claim 8 , further comprising:
sorting, based on attributes of the data, the data within the dataset, the sorting being contingent upon the reordering of the data within the database based on the determined attribute.
12 . The non-transitory computer-readable storage medium of claim 11 , wherein the sorting is performed in relation to the first set of rows.
13 . The non-transitory computer-readable storage medium of claim 11 , wherein the flattening of the reordered data is based on the sorted data.
14 . The non-transitory computer-readable storage medium of claim 8 , further comprising:
formatting the flattened reordered data into a first format, the formatting enabling the storage of the modified dataset.
15 . A device comprising:
a processor configured to:
read a dataset comprising data stored in an unordered manner in a first set of columns and a first set of rows in a database;
determine an attribute for the data of the dataset;
partition the data based on the determined attribute;
reorder the partitioned data based on boundaries of each partition of the data, the reordering performed via a stripe-based alignment of the partitioned data that results in a reduction of a quantity of data files associated with the dataset;
modify the dataset by flattening the reordered data; and
store the modified dataset in the database.
16 . The device of claim 15 , wherein the reordering comprises modifying a configuration of the data within the dataset respective to the first set of columns and first set of rows.
17 . The device of claim 15 , wherein the reordering comprises grouping the data within the dataset based at least on the determined attribute.
18 . The device of claim 15 , wherein the processor is further configured to:
sort, based on attributes of the data, the data within the dataset, the sorting being contingent upon the reordering of the data within the database based on the determined attribute, wherein the flattening of the reordered data is based on the sorted data.
19 . The device of claim 18 , wherein the sorting is performed in relation to the first set of rows.
20 . The device of claim 15 , wherein the processor is further configured to:
format the flattened reordered data into a first format, the formatting enabling the storage of the modified dataset.