Asynchronous methods of data classification using change journals and other data structures
View Patent ↗Aspects of the present invention are generally concerned with systems and methods for generating a database of metadata that describes system data and storage operations. The database of metadata may be referred to herein as a “metabase.” For example, to generate a metabase, a data agent may traverse a file system to obtain certain characteristics of data managed by the file system while substantially simultaneously detecting and recording change notifications. These actions may be performed even if the actions of the data agent are interrupted one or more times during the traversal of the file system. The data agent may process the characteristics and change notifications to generate and update a metabase. Once formed, the metabase may be consulted to determine changes in system data rather than determining the changes by scanning the data files themselves.
1. A method, performed on a server computer, for generating a database of metadata that describes data objects, the method comprising:
interfacing with a file system associated with data stored on a storage device,
wherein the file system provides access to data objects and metadata included within the data stored on the storage device, and
wherein the file system is located on a client computer, external to, but communicatively coupled to, the server computer;
wherein the interfacing includes:
requesting metadata related to the data objects;
receiving the requested metadata; and
storing the received metadata within the database of metadata; and
populating one or more data structures, wherein populating one or more data structures includes—
registering, on the client computer, to receive change notifications from a change notification service;
receiving a change notification from the change notification service of the client computer,
wherein the change notification includes information regarding an interaction with a data object; and
updating the one or more data structures to include information provided by the received change notification,
wherein populating the one or more data structures occurs during substantially the same time as interfacing with the file system located on the client computer.
2. The method of claim 1 , further comprising performing the following for each of the data objects stored on the storage device:
requesting that the file system provide metadata related to each of the data objects;
receiving the requested metadata; and
storing the received metadata in the database of metadata.
3. The method of claim 1 , wherein the file system of the client computer includes a change journal, and wherein the change journal provides the change notifications.
4. The method of claim 1 , wherein received change notifications created outside of a required time reflect interactions with data objects that occur after interfacing with the file system.
5. The method of claim 1 , further comprising examining the one or more data structures to determine if any received change notifications were received outside of a required time.
6. A non-transitory computer-readable medium having instructions to cause at least one processor to generate a database of metadata that describes data objects, comprising:
interfacing with a file system associated with data stored on a storage device,
wherein the file system provides access to data objects and metadata included within the data stored on the storage device,
wherein the file system is located on a client computer, external to, but communicatively coupled to, the server computer, and
wherein the interfacing includes:
requesting metadata related to the data objects;
receiving the requested metadata; and
storing the received metadata within the database of metadata; and,
populating one or more data structures, wherein populating one or more data structures includes—
registering for a change notification service;
receiving a change notification, wherein the change notification includes information regarding an interaction with a data object; and
updating the one or more data structures to include information provided by the received change notification,
wherein populating the one or more data structures occurs during substantially the same time as interfacing with the file system.
7. The computer-readable medium of claim 6 , further comprising performing the following for each of the data objects stored on the storage device:
requesting that the file system provide metadata related to each of the data objects;
receiving the requested metadata; and
storing the received metadata in the database of metadata.
8. The computer-readable medium of claim 6 , wherein the file system of the client computer includes a change journal, and wherein the change journal provides the change notifications.
9. The computer-readable medium of claim 6 , further comprising examining the one or more data structures to determine if any received change notifications were received outside of a required time.
10. One or more non-transitory computer-readable data storage media storing instructions that, when executed by a computer, perform a method for generating a database of metadata for data files stored on a data storage device, the method comprising:
scanning a file system associated with the data storage device, wherein the file system indexes the data files stored on the data storage device;
storing information in a data structure, wherein the information relates to interactions with the data files stored on the data storage device;
identifying interactions with the data files stored on the data storage device that are not reflected by the information stored in the data structure;
updating the data structure with information regarding interactions with the data files stored on the data storage device that are not reflected by the information stored in the data structure; and
based on the information stored in the data structure, updating metadata in a database of metadata.
11. The computer-readable media of claim 10 , further comprising populating the database of metadata substantially simultaneously while the storing the information in the data structure.
12. The computer-readable media of claim 10 , wherein the data interactions that are not reflected by the information stored in the one or more data structures include interactions with data objects that have occurred after beginning the scanning of the file system.
13. A system for generating a database of metadata that describes data objects, the system comprising:
at least one processor;
at least one memory coupled to the processor;
a scanning unit configured to scan a file system associated with a computing device to obtain information, wherein the file system provides access to data stored on a data storage device;
a populating unit configured to populate a database of metadata with the obtained information;
a storing unit configured to store information in one or more data structures, wherein the stored information relates to data interactions with the data stored on the data storage device;
an identifying unit configured to identify data interactions that are not reflected by the information stored in the one or more data structures;
a first updating unit configured to update the one or more data structures with information regarding interactions with data stored on the data storage device that are not reflected by the information stored in the one or more data structures;
an accessing unit configured to access the information stored in the one or more data structures; and
a second updating unit configured to update metadata in the database of metadata based on the accessed information.
14. The system of claim 13 , wherein the means for populating unit populates the database of metadata substantially simultaneously while the storing unit stores information in the one or more data structures.
15. The system of claim 13 , wherein the data interactions that are not reflected by the information stored in the one or more data structures include interactions with data objects that have occurred since the scanning unit begins scanning the file system.