Annotating datasets without redundant copying
Disclosed embodiments are methods, apparatuses, and computer-readable media for annotating distributed data without redundant data copying. In one embodiment, a method is disclosed comprising reading a raw dataset, the raw dataset comprising a first set of columns and a first set of rows; generating an annotation dataset, the annotation dataset comprising a second set of columns and a second set of rows; assigning row identifiers to each row in the second set of rows, the row identifiers aligning the second set of rows with the first set of rows based on the underlying storage of the raw dataset and annotation dataset; and writing the annotation dataset to a distributed storage medium.
1 . A method comprising:
receiving, by a device, information related to a dataset, the information corresponding to an origin path of the dataset and a destination path of the dataset, the origin path comprising metadata for the dataset;
analyzing, by the device, the dataset;
determining, by the device, based on the analysis, to move the metadata to the destination path;
analyzing, by the device, metadata associated with the destination path;
replacing, by the device, based on the analysis of the destination path metadata, values of the destination path metadata with values of the metadata for the dataset, the values of the metadata for the dataset corresponding to an absolute path for the dataset; and
storing, by the device, the dataset in accordance with the destination path.
2 . The method of claim 1 , wherein the information further comprises a referrer path, the referrer path comprising a set of optional dataset paths that reference the origin path.
3 . The method of claim 2 , further comprising:
determining to move the metadata to one of the set of optional dataset paths.
4 . The method of claim 3 , further comprising:
determining an optional path that corresponds to the origin path;
replacing values of the optional path with the values of the metadata for the dataset; and
storing the dataset in accordance with the optional path.
5 . The method of claim 1 , wherein the destination path corresponds to a new location for the dataset and the metadata.
6 . The method of claim 1 , further comprising:
removing the origin path upon the storage of the dataset in accordance with the destination path.
7 . The method of claim 1 , further comprising:
copying the dataset from the origin path; and
storing the copy of the dataset in association with the destination path, wherein the storage of the dataset is based on storage of the copy of the dataset.
8 . The method of claim 1 , wherein the dataset comprises raw data, wherein the analysis of the data comprises determining columns and rows of the raw data.
9 . A non-transitory computer-readable storage medium tangibly encoded with computer-executable instructions, that when executed by a messaging server, perform a method comprising:
receiving, by a device, information related to a dataset, the information corresponding to an origin path of the dataset and a destination path of the dataset, the origin path comprising metadata for the dataset;
analyzing, by the device, the dataset, and determining, based on the analysis, to move the metadata to the destination path;
analyzing, by the device, metadata associated with the destination path;
replacing, by the device, based on the analysis of the destination path metadata, values of the destination path metadata with values of the metadata for the dataset, the values of the metadata for the dataset corresponding to an absolute path for the dataset; and
storing, by the device, the dataset in accordance with the destination path.
10 . The non-transitory computer-readable storage medium of claim 9 , wherein the information further comprises a referrer path, the referrer path comprising a set of optional dataset paths that reference the origin path.
11 . The non-transitory computer-readable storage medium of claim 10 , further comprising:
determining to move the metadata to one of the set of optional dataset paths;
determining an optional path that corresponds to the origin path;
replacing values of the optional path with the values of the metadata for the dataset; and
storing the dataset in accordance with the optional path.
12 . The non-transitory computer-readable storage medium of claim 9 , further comprising:
removing the origin path upon the storage of the dataset in accordance with the destination path.
13 . The non-transitory computer-readable storage medium of claim 9 , further comprising:
copying the dataset from the origin path; and
storing the copy of the dataset in association with the destination path, wherein the storage of the dataset is based on storage of the copy of the dataset.
14 . The non-transitory computer-readable storage medium of claim 9 , wherein the dataset comprises raw data, wherein the analysis of the data comprises determining columns and rows of the raw data.
15 . A device comprising:
a processor configured to:
receive information related to a dataset, the information corresponding to an origin path of the dataset and a destination path of the dataset, the origin path comprising metadata for the dataset;
analyze the dataset, and determine, based on the analysis, to move the metadata to the destination path;
analyze metadata associated with the destination path;
replace, based on the analysis of the destination path metadata, values of the destination path metadata with values of the metadata for the dataset, the values of the metadata for the dataset corresponding to an absolute path for the dataset; and
store the dataset in accordance with the destination path.
16 . The device of claim 15 , wherein the information further comprises a referrer path, the referrer path comprising a set of optional dataset paths that reference the origin path.
17 . The device of claim 16 , wherein the processor is further configured to:
determine to move the metadata to one of the set of optional dataset paths;
determine an optional path that corresponds to the origin path;
replace values of the optional path with the values of the metadata for the dataset; and
store the dataset in accordance with the optional path.
18 . The device of claim 15 , wherein the processor is further configured to:
removing the origin path upon the storage of the dataset in accordance with the destination path.
19 . The device of claim 15 , wherein the processor is further configured to:
copy the dataset from the origin path; and
store the copy of the dataset in association with the destination path, wherein the storage of the dataset is based on storage of the copy of the dataset.
20 . The device of claim 15 , wherein the dataset comprises raw data, wherein the analysis of the data comprises determining columns and rows of the raw data.