Framework for authoring data loaders and data savers
Implementing static loaders and savers for the transfer of local and distributed data containers to and from storage systems can be difficult because there are so many different configurations of output formats, data containers and storage systems. Described herein is an extensible componentized data transfer framework for performant and scalable authoring of data loaders and data savers. Abstracted local and distributed workflows drive selection of plug-ins that can be composed by the framework into particular local or distributed scenario loaders and savers. Reusability and code sparsity are maximized.
1. A system comprising:
at least one processor; and
a memory connected to the at least one processor, wherein the at least one memory and the at least one processor are respectively configured to store and execute instructions, including instructions for causing the system to:
abstract, with a data transfer framework, data transfer operations into a plurality of generic workflows, the data transfer framework dynamically composing components of a particular generic workflow of the plurality of generic workflows into a data transfer executable for a particular data flow scenario, data flow direction, data format to convert from or to, data container and storage system.
2. The system of claim 1 , wherein the data transfer framework is extensible.
3. The system of claim 1 , wherein a data flow scenario is one of local or distributed.
4. The system of claim 1 , wherein the data transfer framework coordinates communications between a plurality of processes in a distributed data transfer scenario.
5. The system of claim 1 , wherein a component of a generic output workflow is at least one of:
a setup component that receives parameters comprising a reference to a data container and returns a modifiable record that indicates workflow state;
a decomposer that partitions a data container into a sequence of blocks;
a formatter that serializes a block of the sequence of blocks and returns a memory buffer comprising the block;
a storage writer that writes the memory stream to a storage system; or
a finalizer that in response to determining that serialization of the sequence of blocks was unsuccessful rolls back partial writes.
6. The system of claim 5 , wherein a component of the generic output workflow is a committer that performs a global commit operation.
7. The system of claim 1 , wherein a component of the generic input workflow is one of:
a setup component that receives parameters and returns metadata describing input data from a source and instructions indicating a portion of the source to read from;
an endpoint component that receives the metadata and the instructions and returns a sequence of records; or
a component that assembles the sequence of records into a data container.
8. A method comprising:
abstracting data transfer operations into a plurality of generic componentized workflows;
receiving parameters identifying aspects of a particular data flow scenario;
selecting components associated with a generic workflow corresponding to the particular data flow scenario; and
dynamically composing the selected components into a data transfer executable.
9. The method of claim 8 , wherein the particular data flow scenario is local.
10. The method of claim 8 , wherein the particular data flow scenario is distributed.
11. The method of claim 8 , wherein the data transfer executable is a data loader that loads data into memory.
12. The method of claim 8 , wherein the data transfer executable is a data saver that saves data to a storage system.
13. The method of claim 8 , further comprising:
reusing components stored in a component library.
14. A computer-readable storage medium comprising computer-readable instructions which when executed cause at least one processor of a computing device to:
invoke a data transfer framework in which data transfer operations are abstracted into a plurality of generic workflows, the data transfer framework dynamically composing components of a particular generic workflow of the plurality of generic workflows into a data transfer executable for a particular data flow scenario, data format to convert from or to, data container and storage system.
15. The computer-readable storage medium of claim 14 , comprising further computer-readable instructions which when executed cause the at least one processor to:
establish a level of indirection between application code and the data transfer framework such that a modification to the data transfer framework is transparent to the application code.
16. The computer-readable storage medium of claim 14 , comprising further computer-readable instructions which when executed cause the at least one processor to:
abstract data transfer operations into a plurality of generic componentized workflows.
17. The computer-readable storage medium of claim 14 , comprising further computer-readable instructions which when executed cause the at least one processor to:
receive parameters identifying aspects of a particular data flow scenario;
select components associated with a generic workflow corresponding to the particular data flow scenario; and
dynamically compose the selected components into a data transfer executable.
18. The computer-readable storage medium of claim 14 , comprising further computer-readable instructions which when executed cause the at least one processor to:
compose a data saver.
19. The computer-readable storage medium of claim 14 , comprising further computer-readable instructions which when executed cause the at least one processor to:
compose a data loader.
20. The computer-readable storage medium of claim 19 , comprising further computer-readable instructions which when executed cause the at least one processor to:
synchronize communications between a plurality of processes in a distributed data transfer scenario.