System and method for dynamic, incremental recommendations within real-time visual simulation
In accordance with various embodiments, described herein is a system (Data Artificial Intelligence system, Data AI system), for use with a data integration or other computing environment, that leverages machine learning (ML, DataFlow Machine Learning, DFML), for use in managing a flow of data (dataflow, DF), and building complex dataflow software applications (dataflow applications, pipelines). In accordance with an embodiment, the system can include a software development component and graphical user interface, referred to herein in some embodiments as a pipeline editor, or Lambda Studio IDE, that provides a visual environment for use with the system, including providing real-time recommendations for performing semantic actions on data accessed from an input HUB, based on an understanding of the meaning or semantics associated with the data.
1. A method for use with a data integration or other computing environment comprising:
processing an accessed data, to perform a metadata analysis of the accessed data, wherein the metadata analysis includes determining a classification of the accessed data;
sending, to a knowledge source of a system, a query for semantic actions enabled for the accessed data, wherein the query indicates the classification of the accessed data;
receiving, from the knowledge source, a response to the query, wherein the response indicates one or more semantic actions enabled for the accessed data and identified based on the classification of the data; and
displaying, at a graphical user interface, selected ones of the semantic actions enabled for the accessed data, for selection and use with the accessed data, including automatically providing or updating a list of the selected ones of the semantic actions enabled for the accessed data, during the processing of the accessed data.
2. The method of claim 1 , further comprising providing a first graphical user interface to receive an instruction from a user to perform the one or more semantic actions on the data accessed from an input source, as part of a dataflow application.
3. The method of claim 2 , further comprising generating, subsequent to accessing the data, a second graphical user interface to display the classification for the accessed data and one or more additional semantic actions, appropriate to the accessed data and a user profile, and displaying the second graphical user interface.
4. The method of claim 1 , wherein the metadata analysis of the accessed data includes identifying one or more samples of the accessed data; applying a machine learning process to determine a category of data within the accessed data; and generating a profile of the accessed data, based on the determined category of data, for use in auto-mapping the accessed data.
5. The method of claim 1 , further comprising accessing a knowledge source to obtain metadata about the at least one of sampled data or accessed data.
6. The method of claim 5 , wherein the knowledge source is a system HUB.
7. The method of claim 1 , wherein the method is performed in a cloud or cloud-based computing environment.
8. A system for providing a software development component for use with a data integration or other computing environment, comprising:
one or more processors operable to:
process an accessed data, to perform a metadata analysis of the accessed data, wherein the metadata analysis includes determining a classification of the accessed data;
send, to a knowledge source of the system, a query for semantic actions enabled for the accessed data, wherein the query indicates the classification of the accessed data;
receive, from the knowledge source, a response to the query, wherein the response indicates one or more semantic actions enabled for the accessed data and identified based on the classification of the data; and
display, at a graphical user interface, selected ones of the semantic actions enabled for the accessed data, for selection and use with the accessed data, including automatically providing or updating a list of the selected ones of the semantic actions enabled for the accessed data, during the processing of the accessed data.
9. The system of claim 8 , further comprising a first graphical user interface to receive an instruction from a user to perform the one or more semantic actions on the data accessed from an input source, as part of a dataflow application.
10. The system of claim 9 , further comprising a second graphical user interface to display the classification for the accessed data and one or more additional semantic actions, appropriate to the accessed data and a user profile.
11. The system of claim 8 , wherein the metadata analysis of the accessed data includes identifying one or more samples of the accessed data; applying a machine learning process to determine a category of data within the accessed data; and generating a profile of the accessed data, based on the determined category of data, for use in auto-mapping the accessed data.
12. The system of claim 8 , further comprising accessing a knowledge source to obtain metadata about the at least one of sampled data or accessed data.
13. The system of claim 12 , wherein the knowledge source is a system HUB.
14. The system of claim 8 , wherein the system is provided in a cloud or cloud-based computing environment.
15. A non-transitory computer readable storage medium, including instructions stored thereon which when read and executed by one or more computers cause the one or more computers to perform a method comprising:
processing an accessed data, to perform a metadata analysis of the accessed data, wherein the metadata analysis includes determining a classification of the accessed data;
sending, to a knowledge source of a system, a query for semantic actions enabled for the accessed data, wherein the query indicates the classification of the accessed data;
receiving, from the knowledge source, a response to the query, wherein the response indicates one or more semantic actions enabled for the accessed data and identified based on the classification of the data; and
displaying, at a graphical user interface, selected ones of the semantic actions enabled for the accessed data, for selection and use with the accessed data, including automatically providing or updating a list of the selected ones of the semantic actions enabled for the accessed data, during the processing of the accessed data.
16. The non-transitory computer readable storage medium of claim 15 , further comprising providing a first graphical user interface to receive an instruction from a user to perform the one or more semantic actions on the data accessed from an input source, as part of a dataflow application.
17. The non-transitory computer readable storage medium of claim 16 , further comprising generating, subsequent to accessing the data, a second graphical user interface to display the classification for the accessed data and one or more additional semantic actions, appropriate to the accessed data and a user profile, and displaying the second graphical user interface.
18. The non-transitory computer readable storage medium of claim 16 , further comprising accessing a knowledge source to obtain metadata about the at least one of sampled data or accessed data.
19. The non-transitory computer readable storage medium of claim 18 , wherein the knowledge source is a system HUB.
20. The non-transitory computer readable storage medium of claim 15 , wherein the metadata analysis of the accessed data includes identifying one or more samples of the accessed data; applying a machine learning process to determine a category of data within the accessed data; and generating a profile of the accessed data, based on the determined category of data, for use in auto-mapping the accessed data.