Apparatus and methods for scanning data in a cloud storage service
Methods and apparati for scanning objects stored in a cloud storage system are disclosed. In an embodiment, the method includes determining at least one object that requires scanning; scanning each of the at least one object, wherein the scanning is performed using at least one scanning engine; and synchronizing the scanning results of the at least one object with a plurality of devices connected to the cloud store system.
1. A method for scanning objects stored in a cloud storage system, the scanning of the objects being for a purpose other than detecting a volume failure issue, comprising:
determining that at least one of the objects stored in the cloud storage system requires scanning;
scanning each of the at least one object, wherein the scanning is performed using at least one scanning engine;
tracking objects that have been scanned by:
assigning a monotonically increasing sequence number to each object;
setting a high watermark value, wherein the high watermark value is a highest sequence number that has been scanned, thereby the high watermark value determines in part if subsequent scanning for the at least one object is required; and
storing the high watermark value;
wherein, in the determining step, only objects having a sequence number higher than the stored high watermark are determined to require scanning; and
synchronizing the at least one object with a plurality of devices connected to the cloud storage system based on the scanning results.
2. The method of claim 1 , wherein the determining if the at least one object containing at least one object requires scanning further comprises:
checking a scanning policy for the at least one object, wherein the scanning policy defines at least a type of the at least one scanning engine.
3. The method of claim 1 , wherein at least one scanning engine is a blocking scan, wherein the blocking scan requires completion of the scanning prior to allowing access to the object.
4. The method of claim 1 , wherein the scanning further comprises:
scanning at least one object using any one of: at least one scanning engine running as a background process and at least one scanning engine running as a foreground process, wherein the foreground process is triggered by an access attempt to the at least one object.
5. The method of claim 1 , wherein the at least one scanning engine is any one of: a full-text indexing engine; an image cataloguing and metadata extraction engine, a content type detection engine, an e-discovery and data mining engine, an unstructured data analytics engine.
6. The method of claim 1 , further comprising:
caching the at least one scanned object based on a hash function computed over any one of: the at least one object and a map of the at least one object.
7. The method of claim 6 , wherein the cached object is stored in a distributed hash table.
8. The method of claim 1 , wherein the at least one object contains unstructured data.
9. A method for scanning objects stored in a cloud storage system, the scanning of the objects being for a purpose other than detecting a volume failure issue, comprising:
determining that at least one of the objects stored in the cloud storage system requires scanning;
scanning each of the at least one object, wherein the scanning is performed using at least one scanning engine; and
synchronizing the at least one object with a plurality of devices connected to the cloud storage system based on the scanning results
wherein the at least one scanning engine is a threat detection engine that employs pattern matching on blocks of the at least one object to determine if the at least one object is a threat, wherein determining that an object is a threat is based on the object containing at least one suspect block, wherein a suspect block contains at least one of a non-trivial prefix or non-trivial suffix match; and
wherein the synchronizing is performed by, when the scanning results is that the least one object is a threat, neutralizing the threat from the at least one object from the cloud store system and any device synchronized with the cloud storage system, wherein neutralizing the threat includes any one of: deleting the object from the cloud storage system and the plurality of devices, quarantining the object at the cloud storage system and the plurality of devices, and replacing the object by a safe version of the object at the cloud storage system and the plurality of devices.
10. The method of claim 9 , wherein the determining if the at least one object containing at least one object requires scanning further comprises:
checking a scanning policy for the at least one object, wherein the scanning policy defines at least a type of the at least one scanning engine.
11. The method of claim 9 , wherein the at least one scanning engine performs a blocking scan, wherein the blocking scan requires completion of the scanning prior to allowing access to the object.
12. The method of claim 9 , wherein the scanning further comprises:
scanning at least one object using any one of: the at least one scanning engine running as a background process and the at least one scanning engine running as a foreground process, wherein the foreground process is triggered by an access attempt to the at least one object.
13. The method of claim 9 , wherein the at least one scanning engine is any one of: a full-text indexing engine; an image cataloguing and metadata extraction engine, a content type detection engine, an e-discovery and data mining engine, an unstructured data analytics engine.
14. The method of claim 9 , further comprising:
caching the at least one scanned object based on a hash function computed over any one of: the at least one object and a map of the at least one object.
15. An apparatus for scanning objects stored in a cloud storage system, the scanning of the objects being for a purpose other than detecting a volume failure issue, comprising:
a processing unit; and
a memory coupled to the processing unit, the memory contains instructions that when executed by the processing unit configures the apparatus to:
determine that at least one of the objects stored in the cloud storage system that requires scanning;
scan each of the at least one object, wherein the scanning is performed using at least one scanning engine;
track objects that have been scanned by:
assign a monotonically increasing sequence number to each object;
set a high watermark value, wherein the high watermark value is a highest sequence number that has been scanned, thereby the high watermark value determines in part if subsequent scanning for the at least one object is required; and
store the high watermark value;
wherein only objects having a sequence number higher than the stored high watermark are determined to require scanning; and
synchronize the at least one object with a plurality of devices connected to the cloud store system based on the scanning results.
16. The method of claim 14 , wherein the cached object is stored in a distributed hash table.
17. The method of claim 15 , wherein the determining if the at least one object containing at least one object requires scanning further comprises:
checking a scanning policy for the at least one object, wherein the scanning policy defines at least a type of the at least one scanning engine.
18. The method of claim 15 , wherein at least one scanning engine is a blocking scan, wherein the blocking scan requires completion of the scanning prior to allowing access to the object.
19. The method of claim 15 , wherein the scanning further comprises:
scanning at least one object using any one of: at least one scanning engine running as a background process and at least one scanning engine running as a foreground process, wherein the foreground process is triggered by an access attempt to the at least one object.
20. The method of claim 15 , further comprising:
caching the at least one scanned object based on a hash function computed over any one of: the at least one object and a map of the at least one object.
21. The method of claim 20 , wherein the cached object is stored in a distributed hash table.
22. An apparatus for scanning objects stored in a cloud storage system, the scanning of the objects being for a purpose other than detecting a volume failure issue, comprising:
a processing unit; and
a memory coupled to the processing unit, the memory contains instructions that when executed by the processing unit configures the apparatus to:
determine that at least one of the objects stored in the cloud storage system that requires scanning;
scan each of the at least one object, wherein the scanning is performed using at least one scanning engine; and
synchronize the at least one object with a plurality of devices connected to the cloud store system based on the scanning results;
wherein the at least one scanning engine is a threat detection engine that employs pattern matching on blocks of the at least one object to determine if the at least one object is a threat, wherein determining that an object is a threat is based on the object containing at least one suspect block, wherein a suspect block contains at least one of a non-trivial prefix or non-trivial suffix match; and
wherein the synchronizing is performed by, when the scanning results is that the least one object is a threat, neutralizing the threat from the at least one object from the cloud store system and any device synchronized with the cloud storage system, wherein neutralizing the threat includes any one of: deleting the object from the cloud storage system and the plurality of devices, quarantining the object at the cloud storage system and the plurality of devices, and replacing the object by a safe version of the object at the cloud storage system and the plurality of devices.
23. The method of claim 22 , wherein the determining if the at least one object containing at least one object requires scanning further comprises:
checking a scanning policy for the at least one object, wherein the scanning policy defines at least a type of the at least one scanning engine.
24. The method of claim 22 , wherein at least one scanning engine is a blocking scan, wherein the blocking scan requires completion of the scanning prior to allowing access to the object.
25. The method of claim 22 , wherein the scanning further comprises:
scanning at least one object using any one of: at least one scanning engine running as a background process and at least one scanning engine running as a foreground process, wherein the foreground process is triggered by an access attempt to the at least one object.
26. The method of claim 22 , further comprising:
caching the at least one scanned object based on a hash function computed over any one of: the at least one object and a map of the at least one object.
27. The method of claim 26 , wherein the cached object is stored in a distributed hash table.