Curating ambiguous data for use in a data pipeline through interaction with a data source
Methods and systems for curating data by a data manager are disclosed. Data may be curated from various data sources before being provided to downstream consumers that may rely on the trustworthiness of the curated data in order to provide desired computer-implemented services. During the data curation process, data curation resources are used to improve the trustworthiness and/or value of the collected data. However, data curation resources (e.g., data curators, computing resources) may be limited and/or insufficient to perform the data curation process as desired, which may result in unusable and/or uncurated (e.g., untrustworthy) data. Thus, the data may be screened for ambiguous values. A potential replacement value for each ambiguous value may be provided to the data source and the data source may indicate whether the potential replacement value should be used in the data pipeline as a final replacement value for the ambiguous value.
1. A method of curating data by a data manager, the method comprising:
making an identification that data obtained from a data source associated with a data pipeline comprises an ambiguous value;
obtaining a potential replacement value for the ambiguous value;
obtaining a final replacement value for the ambiguous value using the potential replacement value and an interaction with an entity associated with the data source that is injected into the method of curating data to assist the data manager, the interaction with the entity only being used when the data manager determines that there are insufficient data curation resources to remedy the ambiguity in the ambiguous value; and
removing the ambiguous value from the data pipeline while populating the data pipeline with the final replacement value to prevent the ambiguous value from negatively impacting operations and functionalities of a data processing system that uses the data in the data pipeline in one or more processes executed by the data processing system for providing computer-implemented services to downstream consumers of the data.
2. The method of claim 1 , wherein the ambiguous value is subject to multiple valid but contrasting interpretations.
3. The method of claim 2 , wherein the potential replacement value is based on one of the multiple valid but contrasting interpretations.
4. The method of claim 1 , wherein obtaining the potential replacement value comprises:
generating an inference using an inference model, the inference being the potential replacement value and comprises a previously missing portion of the data that resulted in the data being identified as the ambiguous data, the previously missing portion being combined with the ambiguous data to obtain the potential replacement value.
5. The method of claim 4 , wherein the data source collects personal information of the entity, and the inference model is trained on the personal information of the entity.
6. The method of claim 1 , wherein obtaining the final replacement value comprises:
presenting the potential replacement value to the data source;
obtaining a response from the data source that is responsive to the presented potential replacement value;
in a first instance of the obtaining where the response indicates agreement with the potential replacement value:
using the potential replacement value as the final replacement value; and
in a second instance of the obtaining where the response indicates disagreement with the potential replacement value:
using a data source supplied value from the response as the final replacement value.
7. The method of claim 1 , further comprising:
providing the computer-implemented services using the final replacement value from the populated data pipeline.
8. A non-transitory machine-readable medium having instructions stored therein, which when executed by a processor, cause the processor to perform operations for curating data by a data manager, the operations comprising:
making an identification that data obtained from a data source associated with a data pipeline comprises an ambiguous value;
obtaining a potential replacement value for the ambiguous value;
obtaining a final replacement value for the ambiguous value using the potential replacement value and an interaction with an entity associated with the data source that is injected into the method of curating data to assist the data manager, the interaction with the entity only being used when the data manager determines that there are insufficient data curation resources to remedy the ambiguity in the ambiguous value; and
removing the ambiguous value from the data pipeline while populating the data pipeline with the final replacement value to prevent the ambiguous value from negatively impacting operations and functionalities of a data processing system that uses the data in the data pipeline in one or more processes executed by the data processing system for providing computer-implemented services to downstream consumers of the data.
9. The non-transitory machine-readable medium of claim 8 , wherein the ambiguous value is subject to multiple valid but contrasting interpretations.
10. The non-transitory machine-readable medium of claim 9 , wherein the potential replacement value is based on one of the multiple valid but contrasting interpretations.
11. The non-transitory machine-readable medium of claim 8 , wherein obtaining the potential replacement value comprises:
generating an inference using an inference model, the inference being the potential replacement value and comprises a previously missing portion of the data that resulted in the data being identified as the ambiguous data, the previously missing portion being combined with the ambiguous data to obtain the potential replacement value.
12. The non-transitory machine-readable medium of claim 11 , wherein the data source collects personal information of the entity, and the inference model is trained on the personal information of the entity.
13. The non-transitory machine-readable medium of claim 8 , wherein obtaining the final replacement value comprises:
presenting the potential replacement value to the data source;
obtaining a response from the data source that is responsive to the presented potential replacement value;
in a first instance of the obtaining where the response indicates agreement with the potential replacement value:
using the potential replacement value as the final replacement value; and
in a second instance of the obtaining where the response indicates disagreement with the potential replacement value:
using a data source supplied value from the response as the final replacement value.
14. The non-transitory machine-readable medium of claim 8 , further comprising:
providing the computer-implemented services using the final replacement value from the populated data pipeline.
15. A data manager, comprising:
a processor; and
a memory coupled to the processor to store instructions, which when executed by the processor, cause the processor to perform operations for curating data by the data manager, the operations comprising:
making an identification that data obtained from a data source associated with a data pipeline comprises an ambiguous value;
obtaining a potential replacement value for the ambiguous value;
obtaining a final replacement value for the ambiguous value using the potential replacement value and an interaction with an entity associated with the data source that is injected into the method of curating data to assist the data manager, the interaction with the entity only being used when the data manager determines that there are insufficient data curation resources to remedy the ambiguity in the ambiguous value; and
removing the ambiguous value from the data pipeline while populating the data pipeline with the final replacement value to prevent the ambiguous value from negatively impacting operations and functionalities of a data processing system that uses the data in the data pipeline in one or more processes executed by the data processing system for providing computer-implemented services to downstream consumers of the data.
16. The data manager of claim 15 , wherein the ambiguous value is subject to multiple valid but contrasting interpretations.
17. The data manager of claim 16 , wherein the potential replacement value is based on one of the multiple valid but contrasting interpretations.
18. The data manager of claim 15 , wherein obtaining the potential replacement value comprises at least one selected from a list consisting of:
applying a set of rules to the data to obtain the potential replacement value;
generating an inference using an inference model, the inference being the potential replacement value; and
performing a similarity analysis for the data to obtain the potential replacement value.
19. The data manager of claim 15 , wherein obtaining the final replacement value comprises:
presenting the potential replacement value to the data source;
obtaining a response from the data source that is responsive to the presented potential replacement value;
in a first instance of the obtaining where the response indicates agreement with the potential replacement value:
using the potential replacement value as the final replacement value; and
in a second instance of the obtaining where the response indicates disagreement with the potential replacement value:
using a data source supplied value from the response as the final replacement value.
20. The data manager of claim 15 , wherein the operations further comprise:
providing the computer-implemented services using the final replacement value from the populated data pipeline.