Method and system for detecting unwanted content of files
View Patent ↗Techniques for detecting unwanted data are described herein. In one embodiment, a request is received for storing a data object in a storage system from a client over a network, where the request includes first representative data representing the data object without including actual content of the data object. It is detected whether the data object contains unwanted content by comparing the first representative data with second representative data without accessing the actual content of the data object, where the second representative data represents the unwanted content. A response is transmitted to the client over the network indicating whether the data object is likely to contain the unwanted object based on comparison of the first and second representative data.
1. A computer-implemented method for detecting unwanted content, the method comprising:
receiving a request for storing a data object in a storage system from a client over a network, the data object comprising a plurality of chunks, each chunk represented by a representative data, the request including a first representative data representing a first chunk of the data object without including actual content of the data object;
determining whether the data object has been stored at the storage system prior to determining whether the data object is likely to contain unwanted content;
in response to determining the data object has not been stored at the storage system, determining whether the data object is likely to contain unwanted content by comparing the first representative data with a second representative data without accessing the actual content of the data object and without comparing all the representative data representing the plurality of chunks of the data object against the second representative, the second representative data representing the unwanted content that should not be stored in the storage system based on a policy configured by an administrator associated with the storage system;
in response to determining the data object has not been stored at the storage system and further in response to determining that the data object is likely to contain unwanted content, transmitting a first response to the client over the network informing the client that the data object is likely to contain the unwanted content based on comparison of the first and second representative data, wherein the first response causes the client not to send the data object;
in response to determining the data object has not been stored at the storage system and further in response to determining the data object is not likely to contain unwanted content, transmitting a second response to the client over the network informing the client that the data object is not likely to contain the unwanted content based on comparison of the first and second representative data, wherein the second response causes the client to send the data object;
receiving the actual data object from the client, wherein the actual data object is received only if the second response was sent to the client indicating that the data object is unlikely to contain the unwanted content; and
storing the data object in the storage system including the first representative data of the data object.
2. The method of claim 1 , wherein the second representative data was generated based on one or more data objects that contain the unwanted content using an identical predetermined algorithm that was used to generate the first representative data.
3. The method of claim 1 , wherein the first representative data is generated based on at least one of one or more features extracted from the data object and a fingerprint computed by hashing the data object.
4. The method of claim 1 , wherein the first representative data was generated by the client using a predetermined function that was used to generate the second representative data.
5. The method of claim 1 , wherein the storage system is a deduplicated storage system having a plurality of storage units, each of the storage units storing data that is deduplicated therein.
6. The method of claim 1 , wherein the first representative data is generated by:
partitioning a file associated with the data object into a plurality of chunks; and
for each of the chunks, generating third representative data, wherein the first representative data is generated based on at least a portion of the third representative data.
7. The method of claim 6 , wherein the third representative data is generated based on at least one of one or more features extracted from each of the chunks and a fingerprint computed by hashing each of the chunks.
8. The method of claim 7 , wherein third representative data is generated by applying a data filtering technique using only a subset of bits of the data.
9. The method of claim 1 , wherein indicating a likelihood that the data object contains unwanted content comprises:
determining an amount of second representative data that matches the first representative data; and
indicating that the data object is likely to contain the unwanted content if the amount of matched second representative data exceeds a predetermined threshold.
10. A non-transitory computer-readable storage medium having instructions stored therein, which when executed by a computer, cause the computer to perform operations comprising:
receiving a request for storing a data object in a storage system from a client over a network, the data object comprising a plurality of chunks, each chunk represented by a representative data, the request including a first representative data representing a first chunk of the data object without including actual content of the data object;
determining whether the data object has been stored at the storage system prior to determining whether the data object is likely to contain unwanted content;
in response to determining the data object has not been stored at the storage system, determining whether the data object is likely to contain unwanted content by comparing the first representative data with a second representative data without accessing the actual content of the data object and without comparing all the representative data representing the plurality of chunks of the data object against the second representative, the second representative data representing the unwanted content that should not be stored in the storage system based on a policy configured by an administrator associated with the storage system;
in response to determining the data object has not been stored at the storage system and further in response to determining that the data object is likely to contain unwanted content, transmitting a first response to the client over the network informing the client that the data object is likely to contain the unwanted content based on comparison of the first and second representative data, wherein the first response causes client not to send the data object;
in response to determining the data object has not been stored at the storage system and further in response to determining the data object is not likely to contain unwanted content, transmitting a second response to the client over the network informing the client that the data object is not likely to contain the unwanted content based on comparison of the first and second representative data, wherein the second response causes the client to send the data object;
receiving the actual data object from the client, wherein the actual data object is received only if the second response was sent to the client indicating that the data object is unlikely to contain the unwanted content; and
storing the data object in the storage system including the first representative data of the data object.
11. The non-transitory computer-readable storage medium of claim 10 , wherein the second representative data was generated based on one or more data objects that contain the unwanted content using an identical predetermined algorithm that was used to generate the first representative data.
12. The non-transitory computer-readable storage medium of claim 10 , wherein the first representative data is generated based on at least one of one or more features extracted from the data object and a fingerprint computed by hashing the data object.
13. The non-transitory computer-readable storage medium of claim 10 , wherein the storage system is a deduplicated storage system having a plurality of storage units, each of the storage units storing data that is deduplicated therein.
14. A system, comprising:
a processor; and
a memory coupled to the processor to store instructions, which when executed from the memory, cause the processor to
receive a request for storing a data object in a storage system from a client over a network, the data object comprising a plurality of chunks, each chunk represented by a representative data, the request including a first representative data representing a first chunk of the data object without including actual content of the data object,
determine whether the data object has been stored at the storage system prior to determining whether the data object is likely to contain unwanted content,
in response to determining the data object has not been stored at the storage system, determine whether the data object is likely to contain unwanted content by comparing the first representative data with a second representative data without accessing the actual content of the data object and without comparing all the representative data representing the plurality of chunks of the data object against the second representative, the second representative data representing the unwanted content that should not be stored in the storage system based on a policy configured by an administrator associated with the storage system,
in response to determining the data object has not been stored at the storage system and further in response to determining that the data object is likely to contain unwanted content, transmit a first response to the client over the network informing the client that the data object is likely to contain the unwanted content based on comparison of the first and second representative data, wherein the response causes the client not to send the data object,
in response to determining the data object has not been stored at the storage system and further in response to determining the data object is not likely to contain unwanted content, transmit a second response to the client over the network informing the client that the data object is not likely to contain the unwanted content based on comparison of the first and second representative data, wherein the second response causes the client to send the data object, and
wherein the actual data object is received from the client if the second response was sent to the client indicating that the data object is unlikely to contain the unwanted content and the data object is stored in the storage system including the first representative data of the data object.
15. The system of claim 14 , wherein the second representative data was generated based on one or more data objects that contain the unwanted content using an identical predetermined algorithm that was used to generate the first representative data.
16. The system of claim 14 , wherein the first representative data is generated based on at least one of one or more features extracted from the data object and a fingerprint computed by hashing the data object.
17. The system of claim 14 , wherein the storage system is a deduplicated storage system having a plurality of storage units, each of the storage units storing data that is deduplicated therein.