Method and apparatus for efficiently searching data in a storage system
View Patent ↗Techniques for searching data in a storage system are described herein. In one embodiment, in response to a request for searching target data in a storage system, first representative data for the target data being searched are generated by applying a predetermined algorithm to at least a portion of the target data. The first representative data are searched and compared with second representative data representing one or more data sets stored in the storage system. It is indicated a likelihood that the target data or similar content has been found in the storage system based on the search and comparison.
1. A computer-implemented method for searching data in a storage system, the method comprising:
receiving a request for searching a target data file in a storage system from a remote source;
in response to the request, partitioning the target data file into a plurality of data segments;
generating first representative data for the data segments of the target data file being searched by applying a predetermined algorithm to at least a portion of the data segments, wherein generating first representative data for the target data file comprises:
partitioning the target data file into a plurality of chunks, and
for each of the chunks, generating third representative data, wherein the first representative data is generated based on at least a portion of the third representative data;
searching and comparing the first representative data with second representative data representing one or more data sets stored in the storage system, wherein the second representative data was previously generated and stored in the storage system when the data sets were being stored in the storage system;
determining a likelihood that all of the plurality of data segments of the target data file have been found in the storage system based on the search and comparison, without accessing the data sets, including:
determining an amount of second representative data that matches the first representative data, and
indicating that the target data file has likely been found in the storage system if the amount of matched second representative data exceeds a predetermined threshold;
in response to determining the amount of matched second representative data exceeds a predetermined threshold:
identifying one or more data sets represented by the matched second representative data, and
comparing the target data file with the identified one or more data sets to confirm that the target data file has been found; and
transmitting a response to the remote source to indicate that the target data file has been found if data segments with matched first representative data match corresponding data sets.
2. The method of claim 1 , wherein the searching is performed based on at least one of files, objects, blocks, chunks, and metadata.
3. The method of claim 1 , wherein the second representative data is searched when actual data represented by the second representative data is accessed in the storage system.
4. The method of claim 1 , wherein the second representative data has been previously generated from the one or more data sets, and wherein the search and comparison are performed without accessing the one or more data sets stored in the storage system.
5. The method of claim 1 , wherein the third representative data is generated based on at least one of one or more features extracted from each of the chunks and a fingerprint computed by hashing each of the chunks.
6. The method of claim 1 , wherein the third representative data is generated such that data similar to the second representative data can be found.
7. The method of claim 6 , wherein generating third representative data comprises applying a data filtering technique using only a subset of bits of the data.
8. The method of claim 1 , further comprising:
identifying one or more data sets represented by the matched second representative data; and
comparing the target data file with the identified one or more data sets to confirm that data similar to the target data file has been found.
9. The method of claim 8 , wherein the data is confirmed when similarity is above a predetermined threshold of data in common.
10. The method of claim 8 , wherein determining similarity comprises applying a filtering function to the target data file and the one or more data sets to exclude a subset of bits of the target data file and the one or more data sets.
11. The method of claim 1 , wherein the storage system is a deduplicated storage system.
12. A non-transitory computer-readable storage medium having instructions stored therein, which when executed by a computer, cause the computer to perform operations comprising:
receiving a request for searching a target data file in a storage system from a remote source;
in response to the request, partitioning the target data file into a plurality of data segments;
generating first representative data for the data segments of the target data file being searched by applying a predetermined algorithm to at least a portion of the data segments, wherein generating first representative data for the target data file comprises:
partitioning the target data file into a plurality of chunks, and
for each of the chunks, generating third representative data, wherein the first representative data is generated based on at least a portion of the third representative data;
searching and comparing the first representative data with second representative data representing one or more data sets stored in the storage system, wherein the second representative data was previously generated and stored in the storage system when the data sets were being stored in the storage system;
determining a likelihood that all of the plurality of data segments of the target data file have been found in the storage system based on the search and comparison, without accessing the data sets, including:
determining an amount of second representative data that matches the first representative data, and
indicating that the target data file has likely been found in the storage system if the amount of matched second representative data exceeds a predetermined threshold;
in response to determining the amount of matched second representative data exceeds a predetermined threshold:
identifying one or more data sets represented by the matched second representative data, and
comparing the target data file with the identified one or more data sets to confirm that the target data file has been found; and
transmitting a response to the remote source to indicate that the target data file has been found if data segments with matched first representative data match corresponding data sets.
13. The non-transitory computer-readable storage medium of claim 12 , wherein the second representative data has been previously generated from the one or more data sets, and wherein the search and comparison are performed without accessing the one or more data sets stored in the storage system.
14. The non-transitory computer-readable storage medium of claim 12 , wherein the second representative data is generated from the one or more data sets by accessing the one or more data sets stored in the storage system.
15. A storage system, comprising:
a processor;
a memory coupled to the processor;
a representative data generator executed from the memory by the processor, to receive a request for searching a target data file in the storage system from a remote source, in response to the request, to partition the target data file into a plurality of data segments and to generate first representative data for the data segments of the target data file being searched by applying a predetermined algorithm to a portion of the data segments, wherein generating first representative data for the target data file comprises partitioning the target data file into a plurality of chunks, and for each of the chunks, generating third representative data, wherein the first representative data is generated based on at least a portion of the third representative data; and
a search engine to search and compare the first representative data with second representative data representing one or more data sets stored in the storage system, to determine a likelihood that all of the plurality of data segments of the target data file have been found in the storage system based on the search and comparison, without accessing the data sets, including determine an amount of second representative data that matches the first representative data, and indicate that the target data file has likely been found in the storage system if the amount of matched second representative data exceeds a predetermined threshold, in response to determining the amount of matched second representative data exceeds a predetermined threshold, identify one or more data sets represented by the matched second representative data, and compare the target data file with the identified one or more data sets to confirm that the target data file has been found, and transmit a response to the remote source to indicate that the target data file has been found if data segments with matched first representative data match corresponding data sets.
16. The system of claim 15 , wherein the second representative data has been previously generated from the one or more data sets, and wherein the search and comparison are performed without accessing the one or more data sets stored in the storage system.
17. The system of claim 15 , wherein the second representative data is generated from the one or more data sets by accessing the one or more data sets stored in the storage system.