Partial file restore in a data storage system
View Patent ↗The data storage system according to certain aspects can implement partial file restore, where only a portion of the secondary copy of a file is restored. Such portion may be designated by one or more application offsets for the file. The system may provide an in-chunk index that includes mapping information between the application offsets and the secondary copy offsets. Chunks may refer to logical data units in which secondary copies are stored, and the in-chunk index for a chunk may be stored in secondary storage with the chunk. Because the mapping information may not be provided at a fixed interval, the system can search through application offsets in the in-chunk index to locate the secondary copy offset corresponding to the portion application offset(s). In this manner, the system may restore the designated portion of the secondary copy in a fast and efficient manner by using the in-chunk index.
1. A method of storing files in secondary storage in a data storage system, comprising:
using one or more computing devices comprising computer hardware:
initiating copying of a plurality of files from a primary storage subsystem to a secondary storage subsystem, wherein data stored on the secondary storage subsystem is stored in one or more chunks, and each chunk is a logical data unit for storing the data in the secondary storage subsystem in one or more secondary storage devices residing in the secondary storage subsystem;
copying a first portion of a first file of the plurality of files from the primary storage subsystem to a buffer for writing to the secondary storage subsystem;
creating a first entry in an index for a first chunk of the one or more chunks, the index stored in association with the first chunk in the secondary storage subsystem, the first entry corresponding to the first portion of the first file and comprising:
a first application offset determined by a software application that accessed the first file and that corresponds to the first portion of the first file, wherein the first application offset designates a starting position within the first file of the first portion of the first file to be restored from a secondary copy of the first file in the first chunk stored in the secondary storage subsystem; and
a first secondary storage offset indicating a location of the first portion of the first file within the secondary copy of the first file in the first chunk in the secondary storage subsystem;
copying a second portion of the first file from the primary storage subsystem to the buffer for writing to the secondary storage subsystem;
creating a second entry in the index for the first chunk, the second entry corresponding to the second portion of the first file and comprising:
a second application offset determined by the software application that accessed the first file and that corresponds to the second portion of the first file, wherein the second application offset designates a starting position within the first file of the second portion of the first file to be restored from the secondary copy of the first file in the first chunk stored in the secondary storage subsystem; and
a second secondary storage offset indicating a location of the second portion of the first file within the secondary copy of the first file in the first chunk in the secondary storage subsystem;
writing the first portion from the buffer to the location indicated by the first secondary storage offset and writing the first entry to the first chunk in response to the first portion being written from the buffer; and
writing the second portion from the buffer to the location indicated by the second secondary storage offset and writing the second entry to the first chunk in response to the second portion being written from the buffer,
wherein creation of the secondary copy involves a series of transactions in which data is written to the buffer and then written from the buffer to the secondary storage subsystem, and wherein an amount of data written to the buffer in each transaction is not predetermined.
2. The method of claim 1 , wherein the index is contained within the first chunk.
3. The method of claim 1 , wherein each of the one or more chunks is of equal size.
4. The method of claim 1 , wherein the first entry is written to the secondary storage subsystem substantially concurrently with the writing of the first portion of the first file to the first chunk, and the second entry is written to the secondary storage subsystem substantially concurrently with the writing of the second portion of the first file to the first chunk.
5. The method of claim 1 , further comprising:
copying a third portion of the first file from the primary storage subsystem to the buffer for writing to the secondary storage subsystem; and
creating a third entry in the index for the first chunk, the third entry corresponding to the third portion of the first file and comprising:
a third application offset determined by the software application that accessed the first file and corresponding to the third portion; and
a third secondary storage offset indicating a location of the third portion within the secondary copy of the first file in the first chunk in the secondary storage subsystem,
wherein a first interval between the first application offset and the second application offset is different from a second interval between the second application offset and the third application offset.
6. The method of claim 5 , wherein a third interval between the first secondary storage offset and the second secondary storage offset is different from a fourth interval between the second secondary storage offset and the third secondary storage offset.
7. The method of claim 6 , wherein said writing the first portion to the location indicated by the first secondary storage offset comprises deduplicating or compressing the first portion.
8. The method of claim 1 , wherein the first chunk comprises metadata information relating to the first file stored in association with the first file.
9. The method of claim 1 , further comprising:
copying a third portion of the first file from the primary storage subsystem to the buffer for writing to the secondary storage subsystem;
creating a third entry in a second index for a second chunk of the one or more chunks, the second index stored in association with the second chunk, the third entry corresponding to the third portion of the first file and comprising:
a third application offset determined by the software application that accessed the first file and corresponding to the third portion;
a third secondary storage offset indicating a location of the third portion within the secondary copy of the first file in the second chunk in the secondary storage subsystem; and
corresponding byte position information relating to a byte position of the third portion within the second chunk; and
writing the third portion to the location indicated by the third second storage offset.
10. A data storage system for storing files in secondary storage, comprising:
a storage manager executing on computer hardware and configured to:
initiate copying of a plurality of files from primary storage subsystem to a secondary storage subsystem, wherein data stored on the secondary storage subsystem is stored in one or more chunks, and each chunk is a logical data unit for storing the data in the secondary storage subsystem in one or more secondary storage devices residing in the secondary storage subsystem; and
one or more computing devices comprising computer hardware and configured to:
copy a first portion of the first file of the plurality of files from the primary storage subsystem to a buffer for writing to the secondary storage subsystem;
create a first entry in an index for a first chunk of the one or more chunks, the index stored in association with the first chunk in the secondary storage subsystem, the first entry corresponding to the first portion of the first file and comprising:
a first application offset determined by a software application that accessed the first file and that corresponds to the first portion of the first file, wherein the first application offset designates a starting position within the first file of the first portion of the first file to be restored from a secondary copy of the first file in the first chunk stored in the secondary storage subsystem; and
a first secondary storage offset indicating a location of the first portion of the first file within the secondary copy of the first file in the first chunk in the secondary storage subsystem;
copy a second portion of the first file from the primary storage subsystem to the buffer for writing to the secondary storage subsystem;
create a second entry in the index for the first chunk, the second entry corresponding to the second portion of the first file and comprising:
a second application offset determined by the software application that accessed the first file and that corresponds to the second portion of the first file, wherein the second application offset designates a starting position within the first file of the second portion of the first file to be restored from the secondary copy of the first file in the first chunk stored in the secondary storage subsystem; and
a second secondary storage offset indicating a location of the second portion within the secondary copy of the first file in the first chunk in the secondary storage subsystem;
write the first portion from the buffer to the location indicated by the first secondary storage offset and write the first entry to the first chunk in response to the first portion being written from the buffer; and
write the second portion from the buffer to the location indicated by the second secondary storage offset and write the second entry to the first chunk in response to the second portion being written from the buffer,
wherein creation of the secondary copy involves a series of transactions in which data is written to the buffer and then written from the buffer to the secondary storage subsystem, and wherein an amount of data written to the buffer in each transaction is not predetermined.
11. The system of claim 10 , wherein the index is contained within the first chunk.
12. The system of claim 10 , wherein each of the one or more chunks is of equal size.
13. The system of claim 10 , wherein the first entry is written to the secondary storage subsystem substantially concurrently with the writing of the first portion of the first file to the first chunk, and the second entry is written to the secondary storage subsystem substantially concurrently with the writing of the second portion of the first file to the first chunk.
14. The system of claim 10 , wherein the one or more computing devices are further configured to:
copy a third portion of the first file from the primary storage subsystem to the buffer for writing to the secondary storage subsystem; and
create a third entry in the index for the first chunk, the third entry corresponding to the third portion of the first file and comprising:
a third application offset determined by the software application that accessed the first file and corresponding to the third portion; and
a third secondary storage offset indicating a location of the third portion within the secondary copy of the first file in the first chunk in the secondary storage subsystem,
wherein a first interval between the first application offset and the second application offset is different from a second interval between the second application offset and the third application offset.
15. The system of claim 14 , wherein a third interval between the first secondary storage offset and the second secondary storage offset of the secondary copy is different from a fourth interval between the second secondary storage offset and the third secondary storage offset.
16. The system of claim 10 , wherein the writing of the first portion to the location indicated by the first secondary storage offset is performed at least in part by deduplicating or compressing the first portion.
17. The system of claim 10 , wherein the first chunk comprises metadata information relating to the first file stored in association with the first file.
18. The system of claim 10 , wherein the one or more computing devices are further configured to:
copy a third portion of the first file from the primary storage subsystem to the buffer for writing to the secondary storage subsystem;
create a third entry in a second index for a second chunk of the one or more chunks, the second index stored in association with the second chunk, the third entry corresponding to the third portion of the first file and comprising:
a third application offset determined by the software application that accessed the first file and corresponding to the third portion;
a third secondary storage offset indicating a location of the third portion within the secondary copy of the first file in the second chunk in the secondary storage subsystem; and
corresponding byte position information relating to a byte position of the third portion within the second chunk; and
write the third portion to the location indicated by the third secondary storage offset within the second chunk.