System and method for combining data streams in pipelined storage operations in a storage network
View Patent ↗Described herein are systems and methods for multiplexing pipelined data for backup operations. Various data streams are combined such as by multiplexing by a multiplexing module. The multiplexing module combines the data from the various data streams received by receiver module(s) into a single stream of chunks. The multiplexing module may combine data from multiple archive files into a single chunk. Additional modules perform other operations on the chunks of data to be transported such as encryption, compression, etc. The data chunks are transmitted via a transport channel to a receive pipeline that includes a second receiver module and other modules. The data chunks are then stored in a backup medium. The chunks are later retrieved and separated such as by demultiplexing for restoring to a client or for further storage as auxiliary copies of the separated data streams or archive files.
1. A method for performing a backup operation, the method comprising:
receiving a first data stream having first data, the first data being obtained by a first application-specific data agent;
receiving a second data stream having second data, the second data being obtained by a second application-specific data agent;
combining the first and second data streams into a single stream of one or more data chunks of an archive file, including writing the first data from the first data stream and the second data from the second data stream into a first data chunk of the one or more data chunks;
transmitting the one or more data chunks over a transport channel to a backup medium; and
storing the one or more data chunks on the backup medium.
2. The method of claim 1 , wherein combining the data streams comprises multiplexing the data streams.
3. The method of claim 1 , wherein said combining the data streams further comprises writing data from a single archive file into more than one data chunk.
4. The method of claim 1 , comprising inserting a plurality of tag headers into the first data chunk, each tag header describing data written in the first data chunk from a corresponding data stream.
5. The method of claim 1 , wherein said combining the data streams further comprises writing data into the first data chunk until the end of an archive file has been reached.
6. The method of claim 1 , wherein the first and second data streams contain data from a plurality of archive files, the method further comprising retrieving the first data chunk from the backup medium and separating data from at least one of the archive files from the first data chunk.
7. The method of claim 6 , comprising separating the data from the archive files written into the first data chunk from each other into separate archive file portions.
8. The method of claim 1 , wherein combining the data streams comprises writing data into the first data chunk until the first data chunk reaches a predetermined size.
9. The method of claim 1 , wherein combining the data streams comprises writing data into the first data chunk until a configurable time interval has lapsed.
10. The method of claim 1 , comprising retrieving the first data chunk from the backup medium.
11. The method of claim 10 , comprising separating the data from at least one of the first and second data streams from the first data chunk.
12. The method of claim 11 , comprising separating the data from all data streams written into the first data chunk from each other into separate data stream portions.
13. The method of claim 11 , wherein combining the data streams comprises multiplexing the data streams and wherein separating comprises demultiplexing the data streams written into the first data chunk.
14. The method of claim 1 , wherein the first data and the second data comprise different types of data, and wherein each of the first and second data comprises at least one of the following types of data: migration data, snapshot data, backup data and archive data.
15. The method of claim 1 , wherein said receiving the first data stream and receiving the second data stream comprise receiving the first and second data in the same buffer.
16. The method of claim 1 , wherein the first data is obtained from a tape medium.
17. A system for performing a backup operation on a plurality of data streams containing data to be backed up, the system comprising:
a first agent module for obtaining first data, from a first application, for transmission in a first data stream;
a second agent module for obtaining second data, from a second application, for transmission in a second data stream, the second application being different than the first application;
one or more receivers for receiving the first data stream and the second data stream;
a multiplexer for combining the first and second data streams into a combined data stream;
a data writer for writing data from the combined data stream portion of the combined first and second data streams into one data chunk, the data chunk having both first data from the first application and second data from the second application; and
one or more backup media for storing the data chunk.
18. The system of claim 17 , comprising a transport channel for transporting the data chunk from the data writer to the backup media.
19. The system of claim 17 , wherein the first agent module and the data writer are part of the same local computing system such that the first data stream is not transmitted over an area network.
20. The system of claim 19 , wherein the first data stream is transmitted via a bus.
21. The system of claim 17 , wherein the first and second data streams are from the same client computer.