Lazy pluggable file handling
A data processing pipeline is configured to create and utilize a file view—an abstract representation of a data file that includes a data file content handle and metadata comprising file properties and pipeline-definable tags—that is passed from stage to stage in the pipeline using out-of-band processing (i.e., processing that is independent from data file record processing). The stages in the pipeline are configured to transform the file view by modifying the metadata to enable efficient data file record processing while minimizing the amount of content that needs to be pulled from the data file and held in memory between stages.
1 . A computer-implemented method for operating a data processing pipeline, comprising:
setting up a pipeline having a plurality of successive processing stages including at least an input stage and an output stage, each stage using in-band lazy processing to generate an in-band processing pipeline output that is utilized as an input to a successive stage, the pipeline operating in a resource-constrained computing environment;
receiving a plurality of data files at the input stage, each of the data files comprising records;
creating a file view for each of the plurality of data files, each file view including metadata and a file handle for accessing content in a respective data file;
passing the file view of each of the plurality of data files through each successive stage in the pipeline using out-of-band processing performed separately from the in-band lazy processing;
operating the pipeline to transform one or more file views by one or more of the stages, each transformation comprising one or more of modifying file content, modifying the metadata, modifying the file handle, or providing a new file handle; and
outputting, from the output stage, an out-of-band stream of transformed file views that the processing stages use to obtain data file information, using in-band lazy processing, without parsing data file content, to minimize processor usage in the resource-constrained computing environment.
2 . The computer-implemented method of claim 1 further including passing a transformed file view from a preceding stage to a successive stage and pulling content through the successive stage of the pipeline using a file handle from the passed transformed file view, wherein the content pulling is performed lazily to minimize memory usage in the resource-constrained computing environment.
3 . The computer-implemented method of claim 2 in which the passing of the transformed file view from the preceding to the successive stage is performed using out-of-band processing.
4 . The computer-implemented method of claim 1 further including generating a plurality of transformed data files from the plurality of transformed file views.
5 . The computer-implemented method of claim 1 in which additional stages are plugged into the pipeline to provide additional processing functionality to the pipeline.
6 . The computer-implemented method of claim 1 in which the metadata comprises file properties and pipeline-definable tags, and in which the modification comprises writing one or more new values to the file properties or the pipeline-definable tags.
7 . One or more hardware-based non-transitory computer-readable memory devices storing computer-executable instructions which, upon execution by one or more processors disposed in a computing device, cause the computing device to:
implement a multi-stage processing pipeline including at least an input stage and an output stage, in which each stage handles data from a data file comprising records received at the input stage in sequence using in-band lazy processing to generate an in-band processing pipeline output, and in which the data file comprises a plurality of records;
generate an abstract representation of the data file as a file view at the input stage, the generating including obtaining file properties of the data file, defining one or more tags that are modifiable by the pipeline stages, and providing a file handle for accessing records in the data file;
pass the file view through each stage of the multi-stage processing pipeline using out-of-band processing performed separately from the in-band lazy processing, in which each stage transforms the file view in sequence to generate a plurality of transformed file views in an out-of-band stream of file views, the file view transformation comprising one or more of updating file properties, modifying the one or more tags, modifying the file handle, or providing a new file handle; and
transform records in the data file in each stage using in-band lazy processing based on a respective transformed file view from the out-of-band stream of file views.
8 . The one or more hardware-based non-transitory computer-readable memory devices of claim 7 in which a stage of the multi-stage processing pipeline transforms the file view by writing a new value to a tag or by writing a new value to the file properties.
9 . The one or more hardware-based non-transitory computer-readable memory devices of claim 7 in which a stage of the multi-stage processing pipeline transforms the file view by creating or modifying a handle by which records that are transformed in the stage are accessed.
10 . The one or more hardware-based non-transitory computer-readable memory devices of claim 7 in which the file view includes metadata that is associated with the data file, the file view being modifiable by a stage of the multi-stage processing pipeline by transforming the metadata.
11 . The one or more hardware-based non-transitory computer-readable memory devices of claim 10 in which each stage of the multi-stage processing pipeline modifies the metadata in the file view as the file view is passed through successive stages in the multi-stage processing pipeline.
12 . The one or more hardware-based non-transitory computer-readable memory devices of claim 10 in which one or more stages of the multi-stage processing pipeline generate new metadata in the file view as the file view is passed through successive stages in the multi-stage processing pipeline.
13 . The one or more hardware-based non-transitory computer-readable memory devices of claim 7 in which the file view is configured with a content file handle with which a stage in the multi-stage processing pipeline accesses records in the data file.
14 . The one or more hardware-based non-transitory computer-readable memory devices of claim 7 in which the file view is used by a stage in the multi-stage processing pipeline to perform pre-processing of the data file without reading the records in the data file.
15 . A computing device, comprising:
at least one processor;
a memory operatively coupled to the at least one processor;
at least one hardware-based non-transitory computer-readable storage device having computer-executable instructions stored thereon which, when executed by the at least one processor, cause the computing device to:
implement a data processing pipeline having multiple stages that successively perform in-band lazy processing of records in a data file in which records are read from the data file and written to the memory;
utilize a file view of the data file, the file view comprising metadata including pipeline-definable tags and further comprising a file handle for reading records in the data file;
pass the file view through each stage of the multi-stage processing pipeline, in which each stage utilizes out-of-band processing, performed separately from the in-band lazy processing, to transform the file view, the out-of-band processing at each stage comprising modifying the metadata and the file handle in the file view, the out-of-band processing being independent from the in-band processing of records in the data file; and
output the transformed file views from the data processing pipeline in an out-of-band stream of file views in which the in-band processing of records in the data file is performed responsively to records being read by a stage using the file handle from a transformed file view in the out-of-band stream of file views.
16 . The computing device of claim 15 in which the data processing pipeline uses the file views to filter or combine records from multiple data files without reading records from the data files.
17 . The computing device of claim 15 in which the metadata further comprises file properties of the data file, the file properties comprising elemental data.
18 . The computing device of claim 17 in which the elemental data comprises one or more of file name, file size, permissions, or creation date.
19 . The computing device of claim 15 in which the multi-stage processing pipeline is instantiated using physical network infrastructure that spans an enterprise network and an edge network.
20 . The computing device of claim 19 in which the multi-stage processing pipeline is instantiated using physical network infrastructure that includes a 4G (4 th generation) or a 5G (5 th generation) mobile network.