Data management techniques using distributed policy agent
Techniques described herein relate to a method for distributed data management. The method may include making a first determination that data is written to a data structure of storage of a data host; obtaining, based on the first determination, data information associated with the data; making a second determination of intent corresponding to the data; generating a global name and metadata corresponding to the data, wherein the metadata comprises the intent; and publishing the global name and the metadata to a global name repository.
1. A method for distributed data management, the method comprising:
receiving data from a local source;
making a first determination that the data is written to a data structure of storage of a data host;
obtaining, based on the first determination, data information associated with the data;
making, based on the data information, a second determination of intent corresponding to the data, wherein the intent comprises an intended use of the data, wherein the second determination is made by evaluating a set of policy rules provided to a monitor agent by a global policy manager;
making a third determination that the intent is not certain;
making, in response to the third determination, a fourth determination of a primary intent using the intent as an input to a machine learning (ML) classifier;
generating a global name and metadata corresponding to the data, wherein the metadata comprises the primary intent; and
publishing the global name and the metadata to a global name repository.
2. The method of claim 1 , wherein the first determination is made by a monitor agent monitoring the data structure.
3. The method of claim 1 , wherein the intent and at least a portion of the data information are obtained via an application programming interface (API) used by an application writing the data.
4. The method of claim 1 , wherein the metadata comprises semantic information comprising a data source identifier, a data generation location, and a timestamp.
5. The method of claim 4 , wherein the semantic information further comprises a sequence identifier.
6. A non-transitory computer readable medium comprising computer readable program code, which when executed by a computer processor enables the computer processor to perform a method for distributed data management, the method comprising:
receiving data from a local source;
making a first determination that the data is written to a data structure of storage of a data host;
obtaining, based on the first determination, data information associated with the data;
making, based on the data information, a second determination of intent corresponding to the data, wherein the intent comprises an intended use of the data, wherein the second determination is made by evaluating a set of policy rules provided to a monitor agent by a global policy manager;
making a third determination that the intent is not certain;
making, in response to the third determination, a fourth determination of a primary intent using the intent as an input to a machine learning (ML) classifier;
generating a global name and metadata corresponding to the data, wherein the metadata comprises the primary intent; and
publishing the global name and the metadata to a global name repository.
7. The non-transitory computer readable medium of claim 6 , wherein the first determination is made by a monitor agent monitoring the data structure.
8. The non-transitory computer readable medium of claim 6 , wherein the intent and at least a portion of the data information are obtained via an application programming interface (API) used by an application writing the data.
9. The non-transitory computer readable medium of claim 6 , wherein the metadata comprises semantic information comprising a data source identifier, a data generation location, and a timestamp.
10. The non-transitory computer readable medium of claim 9 , wherein the semantic information further comprises a sequence identifier.
11. A system for distributed data management, the system comprising:
a monitor agent, executing on a processor comprising circuitry, operatively connected to a services management application and to storage of a data host, and configured to:
receiving data from a local source;
make a first determination that the data is written to a data structure of storage of a data host;
obtain, based on the first determination, data information associated with the data;
make, based on the data information, a second determination of intent corresponding to the data, wherein the intent comprises an intended use of the data, wherein the second determination is made by evaluating a set of policy rules provided to a monitor agent by a global policy manager;
make a third determination that the intent is not certain;
make, in response to the third determination, a fourth determination of a primary intent using the intent as an input to a machine learning (ML) classifier;
generating a global name and metadata corresponding to the data, wherein the metadata comprises the primary intent; and
publish the global name and the metadata to a global name repository.
12. The system of claim 11 , wherein the first determination is made by a monitor agent monitoring the data structure.
13. The system of claim 11 , wherein the intent and at least a portion of the data information are obtained via an application programming interface (API) used by an application writing the data.
14. The system of claim 11 , wherein the metadata comprises semantic information comprising a data source identifier, a data generation location, a timestamp, and a sequence identifier.