Method and system creating and using sub-data confidence fabrics
In general, in one aspect, the invention relates to a method for managing data. The method includes obtaining, by a local data manager, a first data confidence fabric (DCF) configuration file, where the first DCF configuration file is associated with a first DCF pipeline and a first workload. The method further includes registering the first DCF pipeline in a DCF pipeline registry, obtaining a data set, identifying the first DCF pipeline using the DCF pipeline registry, and processing the data set based on the first DCF pipeline to obtain first processed data.
1. A method for managing data, the method comprising:
obtaining, by a local data manager, a first data confidence fabric (DCF) configuration file, wherein the first DCF configuration file is associated with a first DCF pipeline and a first workload;
registering the first DCF pipeline in a DCF pipeline registry;
obtaining a data set;
identifying the first DCF pipeline using the DCF pipeline registry; and
processing the data set based on the first DCF pipeline to obtain first processed data.
2. The method of claim 1 , further comprising:
obtaining, by the local data manager, a second DCF configuration file, wherein the second DCF configuration file is associated with a second DCF pipeline and a second workload;
registering the second DCF pipeline in the DCF pipeline registry; and
processing the data set based on the second DCF pipeline to obtain second processed data.
3. The method of claim 2 , wherein the first processed data and the second processed data are different.
4. The method of claim 2 , wherein the first processed data is not accessible by the second workload.
5. The method of claim 1 , wherein the local data manager obtains the data set from a sensor.
6. The method of claim 1 , further comprising:
obtaining, by the local data manager, a second DCF configuration file, wherein the second DCF configuration file is associated with a second DCF pipeline and a second workload;
registering the second DCF pipeline in the DCF pipeline registry; and
processing a second data set based on the second DCF pipeline to obtain second processed data,
wherein the data set is obtained from a first local data source and the second data set is obtained from a second local data source.
7. The method of claim 6 ,
wherein the first processed data is not accessible by the second workload; and
wherein the first local data source is not accessible by the second workload.
8. The method of claim 1 , further comprising:
tracking resource usage of the local data manager associated with processing the data set using the first DCF pipeline.
9. The method of claim 1 ,
wherein the first workload is executing on one selected from a group consisting of a client, a local data system, and a second local data system,
wherein a local data system and the second local data system are managed by a data management system, and
wherein the local data system comprises the local data manager.
10. A non-transitory computer readable medium comprising computer readable program code, which when executed by a computer processor enables the computer processor to perform a method, the method comprising:
obtaining, by a local data manager, a first data confidence fabric (DCF) configuration file, wherein the first DCF configuration file is associated with a first DCF pipeline and a first workload;
registering the first DCF pipeline in a DCF pipeline registry;
obtaining a data set;
identifying the first DCF pipeline using the DCF pipeline registry; and
processing the data set based on the first DCF pipeline to obtain first processed data.
11. The non-transitory computer readable medium of claim 10 , the method further comprising:
obtaining, by the local data manager, a second DCF configuration file, wherein the second DCF configuration file is associated with a second DCF pipeline and a second workload;
registering the second DCF pipeline in the DCF pipeline registry; and
processing the data set based on the second DCF pipeline to obtain second processed data.
12. The non-transitory computer readable medium of claim 11 , wherein the first processed data and the second processed data are different.
13. The non-transitory computer readable medium of claim 11 , wherein the first processed data is not accessible by the second workload.
14. The non-transitory computer readable medium of claim 10 , wherein the local data manager obtains the data set from a sensor.
15. The non-transitory computer readable medium of claim 10 , the method further comprising:
obtaining, by the local data manager, a second DCF configuration file, wherein the second DCF configuration file is associated with a second DCF pipeline and a second workload;
registering the second DCF pipeline in the DCF pipeline registry; and
processing a second data set based on the second DCF pipeline to obtain second processed data,
wherein the data set is obtained from a first local data source and the second data set is obtained from a second local data source.
16. The non-transitory computer readable medium of claim 15 ,
wherein the first processed data is not accessible by the second workload; and
wherein the first local data source is not accessible by the second workload.
17. The non-transitory computer readable medium of claim 10 , the method further comprising:
tracking resource usage of the local data manager associated with processing the data set using the first DCF pipeline.
18. The non-transitory computer readable medium of claim 10 ,
wherein the first workload is executing on one selected from a group consisting of a client, a local data system, and a second local data system,
wherein a local data system and the second local data system are managed by a data management system, and
wherein the local data system comprises the local data manager.
19. A system, comprising:
a processor;
memory comprising instructions, which when executed by the processor, perform a method, the method comprising:
obtaining, by a local data manager, a first data confidence fabric (DCF) configuration file, wherein the first DCF configuration file is associated with a first DCF pipeline and a first workload;
registering the first DCF pipeline in a DCF pipeline registry;
obtaining a data set;
identifying the first DCF pipeline using the DCF pipeline registry; and
processing the data set based on the first DCF pipeline to obtain first processed data.
20. The system of claim 19 , wherein the method further comprises:
obtaining, by the local data manager, a second DCF configuration file, wherein the second DCF configuration file is associated with a second DCF pipeline and a second workload;
registering the second DCF pipeline in the DCF pipeline registry; and
processing the data set based on the second DCF pipeline to obtain second processed data,
wherein the first processed data and the second processed data are different,
wherein the first processed data is not accessible by the second workload.