Using metadata to detect file changes for resolving computer system errors
Embodiments are directed towards managing and tracking item identification of a plurality of items to determine if an item is a new or existing item, where an existing item has been previously processed. In some embodiments, two or more item identifiers may be generated. In one embodiment, generating the two or more item identifiers may include analyzing the item using a small item size characteristic, a compressed item, or for an identifier collision. The two or more item identifiers may be employed to determine if the item is a new or existing item. In one embodiment, the two or more item identifiers may be compared to a record about an existing item to determine if the item is a new or existing item. If the item is an existing item, then the item may be further processed to determine if the existing item has actually changed.
1 . A method, comprising:
determining one or more characteristics associated with an item in a file system based on an expiration of a current expiration time value determined using an internal clock of a computing system, wherein the one or more characteristics include one or more of a size of the item, a timestamp associated with the item, a permissions attribute associated with the item, or a combination thereof;
based on a pointer to the item in the file system being at an output location in a queue of the computing system, evaluating the item;
detecting, by one or more processors, that a change to the item in the file system exists by comparing the one or more characteristics to computer stored metadata associated with the item, wherein the computer stored metadata represents one or more of a previously determined size of the item, a previously determined timestamp associated with the item, a previously determined permissions attribute associated with the item, or a combination thereof;
responsive to detecting the change to the item in the file system exists based on the comparison of the one or more characteristics to the computer stored metadata associated with the item:
determining, by the one or more processors, a last accessed position of the item;
analyzing, by the one or more processors, data in the item, in at least a portion of the file system, from the last accessed position to a current end of the item;
updating the computer stored metadata to updated metadata based on the analyzing;
updating the current expiration time value to an updated expiration time value for a next change evaluation of the item, the updated expiration time being less than the current expiration time value; and
using the change to the item to detect and/or resolve system errors or to determine opportunities to improve system performance; and
responsive to detecting a change to another item in the file system does not exist:
determining a different time value for a subsequent change evaluation of the another item in the file system, the different time determined, based on a determination that no change to the another item exists, to be more than the expiration value and configured to reduce a frequency of re-evaluation, thereby reducing resource utilization associated with unnecessary item analysis in the file system, and
updating an expiration time value to the different time value for the subsequent change evaluation of the another item in the file system to improve file system operations by dynamically adapting expiration time values, thereby enhancing the file system's efficiency by reducing computer resource utilization for monitoring data and reduce evaluation backlog of items in the file system scheduled to be evaluated by removing the another item in the queue of the computing system waiting to be evaluated for a longer time duration thereby decreasing a number of items waiting to be evaluated.
2 . The method of claim 1 , wherein the updating the computer stored metadata includes updating at least one of the size of the item, the timestamp associated with the item, or the permissions attribute associated with the item.
3 . The method of claim 1 , wherein the detecting the change to the item is further by analyzing the item.
4 . The method of claim 1 , wherein the one or more characteristics are represented in current metadata associated with the item, and the comparing includes comparing the current metadata to the computer stored metadata stored with respect to the item.
5 . The method of claim 1 , wherein the updated expiration time value is determined using the current expiration time value minus a constant time value.
6 . A system comprising:
one or more processors; and
one or more memory devices storing instructions thereon that, when executed by the one or more processors, cause the one or more processors to execute operations comprising:
determining one or more characteristics associated with an item in a file system based on an expiration of a current expiration time value determined using an internal clock of a computing system, wherein the one or more characteristics include one or more of a size of the item, a timestamp associated with the item, a permissions attribute associated with the item, or a combination thereof;
based on a pointer to the item in the file system being at an output location in a queue of the computing system, evaluating the item;
detecting, by the one or more processors, that a change to the item in the file system exists by comparing the one or more characteristics to computer stored metadata associated with the item, wherein the computer stored metadata represents one or more of a previously determined size of the item, a previously determined timestamp associated with the item, a previously determined permissions attribute associated with the item, or a combination thereof;
responsive to detecting the change to the item in the file system exists based on the comparison of the one or more characteristics to the computer stored metadata associated with the item:
determining, by the one or more processors, a last accessed position of the item;
analyzing, by the one or more processors, data in the item, in at least a portion of the file system, from the last accessed position to a current end of the item;
updating the computer stored metadata to updated metadata based on the analyzing;
updating the current expiration time value to an updated expiration time value for a next change evaluation of the item, the updated expiration time being less than the current expiration time value; and
using the change to the item to detect and/or resolve system errors or to determine opportunities to improve system performance; and
responsive to detecting a change to another item in the file system does not exist:
determining a different time value for a subsequent change evaluation of the another item in the file system, the different time determined, based on a determination that no change to the another item exists, to be more than the expiration value and configured to reduce a frequency of re-evaluation, thereby reducing resource utilization associated with unnecessary item analysis in the file system, and
updating an expiration time value to the different time value for the subsequent change evaluation of the another item in the file system to improve file system operations by dynamically adapting expiration time values, thereby enhancing the file system's efficiency by reducing computer resource utilization for monitoring data and reduce evaluation backlog of items in the file system scheduled to be evaluated by removing the another item in the queue of the computing system waiting to be evaluated for a longer time duration thereby decreasing a number of items waiting to be evaluated.
7 . The system of claim 6 , wherein the updating the computer stored metadata includes updating at least one of the size of the item, the timestamp associated with the item, or the permissions attribute associated with the item.
8 . The system of claim 6 , wherein the detecting the change to the item is further by analyzing the item.
9 . The system of claim 6 , wherein the one or more characteristics are represented in current metadata associated with the item, and the comparing includes comparing the current metadata to the computer stored metadata stored with respect to the item.
10 . The system of claim 6 , wherein the updated expiration time value is determined using the current expiration time value minus a constant time value.
11 . One or more non-transitory computer-storage media storing instructions that, when executed by one or more processors, cause the one or more processors to execute operations comprising:
determining one or more characteristics associated with an item in a file system based on an expiration of a current expiration time value determined using an internal clock of a computing system, wherein the one or more characteristics include one or more of a size of the item, a timestamp associated with the item, a permissions attribute associated with the item, or a combination thereof;
based on a pointer to the item in the file system being at an output location in a queue of the computing system, evaluating the item;
detecting, by the one or more processors, that a change to the item in the file system exists by comparing the one or more characteristics to computer stored metadata associated with the item, wherein the computer stored metadata represents one or more of a previously determined size of the item, a previously determined timestamp associated with the item, a previously determined permissions attribute associated with the item, or a combination thereof;
responsive to detecting the change to the item in the file system exists based on the comparison of the one or more characteristics to the computer stored metadata associated with the item:
determining, by the one or more processors, a last accessed position of the item;
analyzing, by the one or more processors, data in the item, in at least a portion of the file system, from the last accessed position to a current end of the item;
updating the computer stored metadata to updated metadata based on the analyzing;
updating the current expiration time value to an updated expiration time value for a next change evaluation of the item, the updated expiration time being less than the current expiration time value; and
using the change to the item to detect and/or resolve system errors or to determine opportunities to improve system performance; and
responsive to detecting a change to another item in the file system does not exist:
determining a different time value for a subsequent change evaluation of the another item in the file system, the different time determined, based on a determination that no change to the another item exists, to be more than the expiration value and configured to reduce a frequency of re-evaluation, thereby reducing resource utilization associated with unnecessary item analysis in the file system, and
updating an expiration time value to the different time value for the subsequent change evaluation of the another item in the file system to improve file system operations by dynamically adapting expiration time values, thereby enhancing the file system's efficiency by reducing computer resource utilization for monitoring data and reduce evaluation backlog of items in the file system scheduled to be evaluated by removing the another item in the queue of the computing system waiting to be evaluated for a longer time duration thereby decreasing a number of items waiting to be evaluated.
12 . The one or more non-transitory computer storage media of claim 11 , wherein the updating the computer stored metadata includes updating at least one of the size of the item, the timestamp associated with the item, or the permissions attribute associated with the item.
13 . The one or more non-transitory computer storage media of claim 11 , wherein the detecting the change to the item is further by analyzing the item.
14 . The one or more non-transitory computer storage media of claim 11 , wherein the one or more characteristics are represented in current metadata associated with the item, and the comparing includes comparing the current metadata to the computer stored metadata stored with respect to the item.