Method and system for data fabric automated configuration for internet of things devices
A method for managing data in accordance with one or more embodiments of the invention includes obtaining, by a data management system, a data confidence fabric (DCF) configuration request from a client, and in response to the DCF configuration request: providing a plurality of DCF configuration options to the client, obtaining acceptance of a portion of the plurality of DCF configuration options, generating a DCF configuration file based on the portion of the plurality of DCF configuration options, wherein the configuration file specifies a DCF pipeline, and deploying the DCF configuration file to a local data manager.
1. A method for managing data, the method comprising:
obtaining, by a data management system, a data confidence fabric (DCF) configuration request from a client; and
in response to the DCF configuration request:
providing a plurality of DCF configuration options to the client;
obtaining acceptance of a portion of the plurality of DCF configuration options;
generating a DCF configuration file based on the portion of the plurality of DCF configuration options, wherein the configuration file specifies a DCF pipeline; and
deploying the DCF configuration file to a local data manager.
2. The method of claim 1 , further comprising:
obtaining, by the local data manager, a data set;
identifying the DCF pipeline; and
storing the data set based on the DCF pipeline.
3. The method of claim 1 , wherein deploying the DCF configuration file to a local data manager comprises:
sending the configuration file to a DCF configuration daemon of the local data manager; and
prompting the DCF configuration daemon to configure a data ingestion manager of the local data manager to perform the DCF pipeline.
4. The method of claim 1 , wherein the DCF pipeline specifies at least one of: a process for augmenting data, a process for modifying data, and a process for storing data.
5. The method of claim 4 , wherein the process for augmenting data comprises at least one of: generating metadata associated with the data and generating a digital signature associated with the data.
6. The method of claim 4 , wherein the process for modifying data comprises at least one of: encrypting the data and compressing the data.
7. The method of claim 4 , wherein the process for storing the data comprises at least one of: storing a copy of the data in a ledger service, storing a copy of metadata associated with the data in an object storage, or generating a ledger entry associated with the data.
8. A non-transitory computer readable medium comprising computer readable program code, which when executed by a computer processor enables the computer processor to perform a method, the method comprising:
obtaining, by a data management system, a data confidence fabric (DCF) configuration request from a client; and
in response to the DCF configuration request:
providing a plurality of DCF configuration options to the client;
obtaining acceptance of a portion of the plurality of DCF configuration options;
generating a DCF configuration file based on the portion of the plurality of DCF configuration options, wherein the configuration file specifies a DCF pipeline; and
deploying the DCF configuration file to a local data manager.
9. The non-transitory computer readable medium of claim 8 , the method further comprising:
obtaining, by the local data manager, a data set;
identifying the DCF pipeline; and
storing the data set based on the DCF pipeline.
10. The non-transitory computer readable medium of claim 8 , wherein deploying the DCF configuration file to a local data manager comprises:
sending the configuration file to a DCF configuration daemon of the local data manager; and
prompting the DCF configuration daemon to configure a data ingestion manager of the local data manager to perform the DCF pipeline.
11. The non-transitory computer readable medium of claim 8 wherein the DCF pipeline specifies at least one of: a process for augmenting data, a process for modifying data, and a process for storing data.
12. The non-transitory computer readable medium of claim 11 , wherein the process for augmenting data comprises at least one of: generating metadata associated with the data and generating a digital signature associated with the data.
13. The non-transitory computer readable medium of claim 11 , wherein the process for modifying data comprises at least one of: encrypting the data and compressing the data.
14. The non-transitory computer readable medium of claim 11 , wherein the process for storing the data comprises at least one of: storing a copy of the data in a ledger service, storing a copy of metadata associated with the data in an object storage, or generating a ledger entry associated with the data.
15. A system, comprising:
a processor;
memory comprising instructions, which when executed by the processor, perform a method, the method comprising:
obtaining, by a data management system, a data confidence fabric (DCF) configuration request from a client; and
in response to the DCF configuration request:
providing a plurality of DCF configuration options to the client;
obtaining acceptance of a portion of the plurality of DCF configuration options;
generating a DCF configuration file based on the portion of the plurality of DCF configuration options, wherein the configuration file specifies a DCF pipeline, and wherein the DCF pipeline specifies at least one of: a process for augmenting data, a process for modifying data, and a process for storing data; and
deploying the DCF configuration file to a local data manager.
16. The system of claim 15 , the method further comprising:
obtaining, by the local data manager, a data set;
identifying the DCF pipeline; and
storing the data set based on the DCF pipeline.
17. The system of claim 15 , wherein deploying the DCF configuration file to a local data manager comprises:
sending the configuration file to a DCF configuration daemon of the local data manager; and
prompting the DCF configuration daemon to configure a data ingestion manager of the local data manager to perform the DCF pipeline.
18. The system of claim 15 , wherein the process for augmenting data comprises at least one of: generating metadata associated with the data and generating a digital signature associated with the data.
19. The system of claim 15 , wherein the process for modifying data comprises at least one of: encrypting the data and compressing the data.
20. The system of claim 15 , wherein the process for storing the data comprises at least one of: storing a copy of the data in a ledger service, storing a copy of metadata associated with the data in an object storage, or generating a ledger entry associated with the data.