Periodically processing data in files identified using checksums
Embodiments are directed towards managing and tracking item identification of a plurality of items to determine if an item is a new or existing item, where an existing item has been previously processed. In some embodiments, two or more item identifiers may be generated. In one embodiment, generating the two or more item identifiers may include analyzing the item using a small item size characteristic, a compressed item, or for an identifier collision. The two or more item identifiers may be employed to determine if the item is a new or existing item. In one embodiment, the two or more item identifiers may be compared to a record about an existing item to determine if the item is a new or existing item. If the item is an existing item, then the item may be further processed to determine if the existing item has actually changed.
1. A method for periodically examining a plurality of files in a folder, comprising:
calculating a checksum on a particular portion of a file among the plurality of files in the folder;
identifying the file by comparing the checksum to known checksums;
based on finding a match of the checksum with one of the known checksums:
determining that the file has been updated by comparing a current file size with a previous file size of the file and analyzing new data in the file;
based on not finding a match of the checksum with one of the known checksums:
adding the checksum to the known checksums; and
analyzing data in the file;
wherein the method is performed by one or more computing devices.
2. The method as recited in claim 1 , wherein a time period between each periodic examination is separately determined for each file among the plurality of files.
3. The method as recited in claim 1 , wherein the folder includes one or more sub-folders.
4. The method as recited in claim 1 , wherein the plurality of files in the folder do not include files in the folder that have been identified as to be ignored.
5. The method as recited in claim 1 , wherein each file among the plurality of files is identified using a file handle.
6. The method as recited in claim 1 , wherein each file among the plurality of files is accessed using an abstract descriptor.
7. The method as recited in claim 1 , wherein new data in the file is determined by comparing a last accessed position associated with the file with a current end of file of the file.
8. A non-transitory computer readable storage medium, storing instructions for periodically examining a plurality of files in a folder, which when executed by one or more processors cause performance of:
calculating a checksum on a particular portion of a file among the plurality of files in the folder;
identifying the file by comparing the checksum to known checksums;
based on finding a match of the checksum with one of the known checksums:
determining that the file has been updated by comparing a current file size with a previous file size of the file and analyzing new data in the file;
based on not finding a match of the checksum with one of the known checksums:
adding the checksum to the known checksums; and
analyzing data in the file.
9. The non-transitory computer readable storage medium as recited in claim 8 , wherein a time period between each periodic examination is separately determined for each file among the plurality of files.
10. The non-transitory computer readable storage medium as recited in claim 8 , wherein the folder includes one or more sub-folders.
11. The non-transitory computer readable storage medium as recited in claim 8 , wherein the plurality of files in the folder do not include files in the folder that have been identified as to be ignored.
12. The non-transitory computer readable storage medium as recited in claim 8 , wherein each file among the plurality of files is identified using a file handle.
13. The non-transitory computer readable storage medium as recited in claim 8 , wherein each file among the plurality of files is accessed using an abstract descriptor.
14. The non-transitory computer readable storage medium as recited in claim 8 , wherein new data in the file is determined by comparing a last accessed position associated with the file with a current end of file of the file.
15. An apparatus for periodically examining a plurality of files in a folder, comprising:
a file status determination device, implemented at least partially in hardware, that calculates a checksum on a particular portion of a file among the plurality of files in the folder;
wherein the file status determination device identifies the file by comparing the checksum to known checksums;
wherein the file status determination device, based on finding a match of the checksum with one of the known checksums:
determines that the file has been updated by comparing a current file size with a previous file size of the file and analyzes new data in the file;
wherein the file status determination device, based on not finding a match of the checksum with one of the known checksums:
adds the checksum to the known checksums; and
analyzes data in the file.
16. The apparatus as recited in claim 15 , wherein a time period between each periodic examination is separately determined for each file among the plurality of files.
17. The apparatus as recited in claim 15 , wherein the folder includes one or more sub-folders.
18. The apparatus as recited in claim 15 , wherein the plurality of files in the folder do not include files in the folder that have been identified as to be ignored.
19. The apparatus as recited in claim 15 , wherein each file among the plurality of files is identified using a file handle.
20. The apparatus as recited in claim 15 , wherein new data in the file is determined by comparing a last accessed position associated with the file with a current end of file of the file.