Method of “outcome driven data exploration” for datasets, business questions, and pipelines based on similarity mapping of business needs and asset use overlap
One example method includes receiving a query that recites a particular question for which a user who originated the query needs an answer, parsing the query to identify the question, identifying information that is responsive to the question, presenting the information to the user in a user-selectable form, and receiving, from the user, a selection of the information. In some cases, the information presented to the user may include one or more datasets, or one or more pipelines.
1. A method, comprising:
receiving, by a data abstraction layer of a data pipeline architecture, a query that recites a particular question which identifies a problem that a user who originated the query needs to resolve;
creating, by a metadata control plane that is operable to communicate with the data abstraction layer, a record of the question;
extracting, by the metadata control plane from the record, information relating to the question;
using, by the metadata control plane, extracted metadata to trigger an analysis of the question;
based on an outcome of the analysis, identifying, by the metadata control plane, information that is relating to the question;
verifying, by a data governance control plane, if the user is permitted to access the information;
obtaining, by the metadata control plane from a data catalog by way of the data abstraction layer, the information that is responsive to the question;
presenting, by the metadata control plane, the information to the user in a user-selectable form; and
increasing a correlation score of the information selected by the user from the user-selectable form, wherein the correlation score determines whether the information is used, by the metadata control plane, to resolve similar future queries.
2. The method as recited in claim 1 , wherein the information presented to the user comprises a dataset and/or a pipeline.
3. The method as recited in claim 1 , further comprising presenting to the user, by the metadata control plane, prior to receipt of the query:
information that identifies a question similar to the question posed by the user; and
any datasets and pipelines that were used to resolve the similar question.
4. The method as recited in claim 1 , wherein the query comprises a business intent generated based on one or more business questions provided to, and answered by, the user, and the business intent indicates a way in which the user intends to use a dataset or pipeline received by the user in response to the query.
5. The method as recited in claim 1 , wherein the query specifies a pipeline that the user requires as an output.
6. The method as recited in claim 1 , further comprising presenting, to the user by the metadata control plane, insights generated as a result of a pipeline execution process.
7. The method as recited in claim 1 , wherein the query does not include any data labels, and identification of the information responsive to the query does not involve the use of data labels.
8. The method as recited in claim 1 , further comprising comparing the question with one or more other question, and the information presented to the user comprises a list of the other question ranked according to their respective similarity to the question, and the information presented to the user further comprises a respective dataset and/or pipeline corresponding to each of the question in the list.
9. The method as recited in claim 1 , further comprising recording the metadata related to the question and to the information presented to the user.
10. The method as recited in claim 9 , wherein the metadata comprises any one or more of dataset access patterns, data security/access rights, metadata concerning a business question, metadata concerning a business intent, and dataset identification information.
11. A non-transitory storage medium having stored therein instructions that are executable by one or more hardware processors to perform operations comprising:
receiving, by a data abstraction layer of a data pipeline architecture, a query that recites a particular question which identifies a problem that a user who originated the query needs to resolve;
creating, by a metadata control plane that is operable to communicate with the data abstraction layer, a record of the question;
extracting, by the metadata control plane from the record, information relating to the question;
verifying, by a data governance control plane, if the user is permitted to access the information;
using, by the metadata control plane, extracted metadata to trigger an analysis of the question;
based on an outcome of the analysis, identifying, by the metadata control plane, information that is responsive to the question;
obtaining, by the metadata control plane from a data catalog by way of the data abstraction layer, the information that is responsive to the question;
presenting, by the metadata control plane, the information to the user in a user-selectable form; and
increasing a correlation score of the information selected by the user from the user-selectable form, wherein the correlation score determines whether the information is used, by the metadata control plane, to resolve similar future queries.
12. The non-transitory storage medium as recited in claim 11 , wherein the information presented to the user comprises a dataset and/or a pipeline.
13. The non-transitory storage medium as recited in claim 11 , wherein the operations further comprise presenting to the user, by the metadata control plane, prior to receipt of the query:
information that identifies a question similar to the question posed by the user; and
any datasets and pipelines that were used to resolve the similar question.
14. The non-transitory storage medium as recited in claim 11 , wherein the query comprises a business intent generated based on one or more business questions provided to, and answered by, the user, and the business intent indicates a way in which the user intends to use a dataset or pipeline received by the user in response to the query.
15. The non-transitory storage medium as recited in claim 11 , wherein the query specifies a pipeline that the user requires as an output.
16. The non-transitory storage medium as recited in claim 11 , wherein the operations further comprise presenting, to the user by the metadata control plane, insights generated as a result of a pipeline execution process.
17. The non-transitory storage medium as recited in claim 11 , wherein the query does not include any data labels, and identification of the information responsive to the query does not involve the use of data labels.
18. The non-transitory storage medium as recited in claim 11 , wherein the operations further comprise comparing the question with one or more other questions, and the information presented to the user comprises a list of the other questions ranked according to their respective similarity to the question, and the information presented to the user further comprises a respective dataset and/or pipeline corresponding to each of the questions in the list.
19. The non-transitory storage medium as recited in claim 11 , further comprising recording the metadata related to the question and to the information presented to the user.
20. The non-transitory storage medium as recited in claim 19 , wherein the metadata comprises any one or more of dataset access patterns, data security/access rights, metadata concerning a business question, metadata concerning a business intent, and dataset identification information.