Flexible and secure transformation of data using stream pipes
View Patent ↗Responsive to a request to retrieve or store a file, a transformation pipeline may be created to efficiently transform file data one unit at a time in memory. The transformation pipeline includes a sequence of transformation streams, each containing a write method, a read method, and a transformation to be applied. The write method moves a unit of data, for instance, from a memory buffer into an associated stream. The read method reads the unit of data from the stream, calls an associated transformation, and passes the unit of data thus transformed to the next stream or a destination. This process is repeated until all desired and/or required transformations such as compression, encryption, tamper protection, conversion, etc. are applied to the unit of data.
1. A system for streamed transformation of data, the system comprising:
a storage device;
a first computing device coupled to the storage device, the first computing device comprising a memory, a processor, at least one non-transitory computer-readable medium, and stored instructions translatable by the processor to perform:
receiving a request from a second computing device to read or write a file;
responsive to the request, creating a transformation pipeline, the transformation pipeline providing a processing sequence and traffic path for streamed data and comprising a sequence of transformation streams corresponding to a set of transformations to be applied to the file, the creating the transformation pipeline comprising instantiating a stream object for each transformation stream of the transformation streams, the stream object including a write method for moving a unit of data into the each transformation stream and a read method for retrieving the unit of data from the each transformation stream, calling a transformation function of the set of transformations to transform the unit of data, and providing the unit of data thus transformed within the each transformation stream to a next transformation stream or, if there are no more transformation streams in the transformation pipeline, to a destination device;
receiving file data for the file streamed from a source device;
buffering the file data from the source device in a memory buffer at the first computing device;
segmenting buffered file data to produce units of file data;
passing and transforming the file data through the transformation pipeline one unit of file data at a time, wherein transforming a unit of file data comprises:
providing the unit of file data to a first transformation stream in the transformation pipeline, the providing performed by a write method associated with the first transformation stream;
applying a transformation associated with the first transformation stream to the unit of file data, the applying performed by a read method associated with the first transformation stream, the applying including the read method calling a transformation function to perform the transformation; and
performing the providing and the applying until all transformations in the set of transformations have been applied to the unit of file data through the sequence of transformation streams in the transformation pipeline; and
sending the unit of file data so transformed by the set of transformations to a destination device.
2. The system of claim 1 , wherein the transformation pipeline is created based on at least one of:
metadata contained in the request;
file metadata associated with the file; or
a transformation plan specifying the set of transformations to apply to the file.
3. The system of claim 1 , wherein creating the transformation pipeline comprises aggregating or bundling a plurality of stream classes in sequence such that a result from a first transformation stream is passed as input to a second transformation stream in the transformation pipeline.
4. The system of claim 1 , wherein the unit of file data so transformed at the first computing device is not persisted at the first computing device.
5. The system of claim 1 , wherein the file data is passed and transformed through the transformation pipeline one unit of file data at a time until at least one of:
reading of the file by the destination device is complete;
writing of the file to the destination device is complete; or
a termination condition is satisfied.
6. The system of claim 5 , wherein the termination condition is associated with a determination by the first computing device of a difference between a newly generated hash associated with the file and a previously stored hash associated with the file.
7. The system of claim 1 , wherein the set of transformations comprises at least one of:
an encryption transformation and the transformation pipeline which comprises an encryption transformation stream;
a compression transformation and the transformation pipeline which comprises a compression transformation stream; or
a tamper detection transformation and the transformation pipeline which comprises a tamper detection transformation stream.
8. A method for streamed transformation of data, the method comprising:
a first computing device receiving a request from a second computing device to read or write a file, the first computing device comprising a memory, a processor, at least one non-transitory computer-readable medium, and stored instructions translatable by the processor;
responsive to the request, the first computing device creating a transformation pipeline, the transformation pipeline providing a processing sequence and traffic path for streamed data and comprising a sequence of transformation streams corresponding to a set of transformations to be applied to the file, the creating the transformation pipeline comprising instantiating a stream object for each transformation stream of the transformation streams, the stream object including a write method for moving a unit of data into the each transformation stream and a read method for retrieving the unit of data from the each transformation stream, calling a transformation function of the set of transformations to transform the unit of data, and providing the unit of data thus transformed within the each transformation stream to a next transformation stream or, if there are no more transformation streams in the transformation pipeline, to a destination device;
the first computing device receiving file data for the file streamed from a source device;
buffering the file data from the source device in a memory buffer at the first computing device;
the first computing device segmenting buffered file data to produce units of file data;
the first computing device passing and transforming the file data through the transformation pipeline one unit of file data at a time, wherein transforming a unit of file data comprises:
providing the unit of file data to a first transformation stream in the transformation pipeline, the providing performed by a write method associated with the first transformation stream;
applying a transformation associated with the first transformation stream to the unit of file data, the applying performed by a read method associated with the first transformation stream, the applying including the read method calling a transformation function to perform the transformation; and
performing the providing and the applying until all transformations in the set of transformations have been applied to the unit of file data through the sequence of transformation streams in the transformation pipeline; and
the first computing device sending the unit of file data so transformed by the set of transformations to a destination device.
9. The method according to claim 8 , wherein the transformation pipeline is created based on at least one of:
metadata contained in the request;
file metadata associated with the file; or
a transformation plan specifying the set of transformations to apply to the file.
10. The method according to claim 8 , wherein creating the transformation pipeline comprises aggregating or bundling a plurality of stream classes in sequence such that a result from a first transformation stream is passed as input to a second transformation stream in the transformation pipeline.
11. The method according to claim 8 , further comprising:
repeating the passing and transforming the file data through the transformation pipeline until at least one of:
reading of the file by the destination device is complete;
writing of the file to the destination device is complete; or
a termination condition is satisfied.
12. The method according to claim 11 , wherein the termination condition is associated with a determination by the first computing device of a difference between a newly generated hash associated with the file and a previously stored hash associated with the file.
13. A computer program product comprising at least one non-transitory computer-readable medium storing instructions translatable by a processor of a first computing device to perform:
receiving a request from a second computing device to read or write a file;
responsive to the request, creating a transformation pipeline, the transformation pipeline providing a processing sequence and traffic path for streamed data and comprising a sequence of transformation streams corresponding to a set of transformations to be applied to the file, the creating the transformation pipeline comprising instantiating a stream object for each transformation stream of the transformation streams, the stream object including a write method for moving a unit of data into the each transformation stream and a read method for retrieving the unit of data from the each transformation stream, calling a transformation function of the set of transformations to transform the unit of data, and providing the unit of data thus transformed within the each transformation stream to a next transformation stream or, if there are no more transformation streams in the transformation pipeline, to a destination device;
receiving file data for the file streamed from a source device;
buffering the file data from the source device in a memory buffer at the first computing device;
segmenting buffered file data to produce units of file data;
passing and transforming the file data through the transformation pipeline one unit of file data at a time, wherein transforming a unit of file data comprises:
providing the unit of file data to a first transformation stream in the transformation pipeline, the providing performed by a write method associated with the first transformation stream;
applying a transformation associated with the first transformation stream to the unit of file data, the applying performed by a read method associated with the first transformation stream, the applying including the read method calling a transformation function to perform the transformation; and
performing the providing and the applying until all transformations in the set of transformations have been applied to the unit of file data through the sequence of transformation streams in the transformation pipeline; and
sending the unit of file data so transformed by the set of transformations to a destination device.
14. The computer program product of claim 13 , wherein the transformation pipeline is created based on at least one of:
metadata contained in the request;
file metadata associated with the file; or
a transformation plan specifying the set of transformations to apply to the file.
15. The computer program product of claim 13 , wherein creating the transformation pipeline comprises aggregating or bundling a plurality of stream classes in sequence such that a result from a first transformation stream is passed as input to a second transformation stream in the transformation pipeline.
16. The computer program product of claim 13 , wherein the file data is passed and transformed through the transformation pipeline one unit of file data at a time until at least one of:
reading of the file by the destination device is complete;
writing of the file to the destination device is complete; or
a termination condition is satisfied.
17. The computer program product of claim 16 , wherein the termination condition is associated with a determination by the first computing device of a difference between a newly generated hash associated with the file and a previously stored hash associated with the file.