GENERATING FULL METADATA FROM PARTIAL DISTRIBUTED METADATA

1 . A method comprising:

receiving, by a processor, data to write to disk, the data comprising a subset of a dataset;

writing, by the processor, a first portion of the data to disk;

detecting, by the processor, a split boundary after writing the first portion;

recording, by the processor, metadata describing the split boundary;

continuing, by the processor, to write a remaining portion of the data to disk; and

after completing the writing of the data to disk:

generating, by the processor, a partial metadata file for the data, the partial metadata file including the split boundary, and

transmitting, by the processor, the partial metadata to a partial metadata collector.

2 . The method of claim 1 further comprising generating alignment data after recording the split boundary, the alignment data comprising metadata aligning the first portion of the data to a root dataset.

3 . The method of claim 1 , further comprising:

receiving, by the processor, the partial metadata file and a plurality of additional partial metadata files;

sorting, by the processor, the partial metadata file and the plurality of additional partial metadata files to generate a sorted list of partial metadata files;

sorting, by the processor, splits located in each file in the sorted list of partial metadata files; and

writing, by the processor, the sorted list of partial metadata files to disk as a full metadata file.

4 . The method of claim 3 , further comprising validating alignment of the splits after sorting the splits.

5 . The method of claim 1 , the recording metadata describing the split boundary comprising reporting a row count of the split.

6 . The method of claim 1 , the detecting the split boundary comprising detecting that a current file is too large to fit in a memory coupled to the processing device.

7 . The method of claim 1 , the generating the partial metadata file for the data comprising writing a schema to the partial metadata file.

8 . A non-transitory computer readable storage medium for tangibly storing computer program instructions capable of being executed by a computer processor, the computer program instructions defining the steps of:

receiving data to write to disk, the data comprising a subset of a dataset;

writing a first portion of the data to disk;

detecting a split boundary after writing the first portion;

recording metadata describing the split boundary;

continuing to write a remaining portion of the data to disk; and

after completing the writing of the data to disk:

generating a partial metadata file for the data, the partial metadata file including the split boundary, and

transmitting the partial metadata to a partial metadata collector.

9 . The non-transitory computer readable storage medium of claim 8 , the computer program instructions further defining the step of generating alignment data after recording the split boundary, the alignment data comprising metadata aligning the first portion of the data to a root dataset.

10 . The non-transitory computer readable storage medium of claim 8 , the computer program instructions further defining the steps of:

receiving the partial metadata file and a plurality of additional partial metadata files;

sorting the partial metadata file and the plurality of additional partial metadata files to generate a sorted list of partial metadata files;

sorting splits located in each file in the sorted list of partial metadata files; and

writing the sorted list of partial metadata files to disk as a full metadata file.

11 . The non-transitory computer readable storage medium of claim 10 , the computer program instructions further defining the step of validating alignment of the splits after sorting the splits.

12 . The non-transitory computer readable storage medium of claim 8 , the recording metadata describing the split boundary comprising reporting a row count of the split.

13 . The non-transitory computer readable storage medium of claim 8 , the detecting the split boundary comprising detecting that a current file is too large to fit in a memory coupled to the processing device.

14 . The non-transitory computer readable storage medium of claim 8 , the generating the partial metadata file for the data comprising writing a schema to the partial metadata file.

15 . An apparatus comprising:

a processor;

a storage medium for tangibly storing thereon program logic for execution by the processor, the stored program logic causing the processor to perform the operations of:

receiving data to write to disk, the data comprising a subset of a dataset,

writing a first portion of the data to disk,

detecting a split boundary after writing the first portion,

recording metadata describing the split boundary,

continuing to write a remaining portion of the data to disk, and

after completing the writing of the data to disk: generating a partial metadata file for the data, the partial metadata file including the split boundary, and transmitting the partial metadata to a partial metadata collector.

16 . The apparatus of claim 15 the stored program logic further causing the processor to perform the operations of generating alignment data after recording the split boundary, the alignment data comprising metadata aligning the first portion of the data to a root dataset.

17 . The apparatus of claim 15 , the stored program logic further causing the processor to perform the operations of:

receiving the partial metadata file and a plurality of additional partial metadata files;

sorting the partial metadata file and the plurality of additional partial metadata files to generate a sorted list of partial metadata files;

sorting splits located in each file in the sorted list of partial metadata files; and

writing the sorted list of partial metadata files to disk as a full metadata file.

18 . The apparatus of claim 17 , the stored program logic causing the processor to perform the operation of validating alignment of the splits after sorting the splits.

19 . The apparatus of claim 17 , the recording metadata describing the split boundary comprising reporting a row count of the split.

20 . The apparatus of claim 17 , the detecting the split boundary comprising detecting that a current file is too large to fit in a memory coupled to the processing device.