System and methods for performing automatic data aggregation
Systems, apparatuses, and methods for automated data aggregation, automated webpage navigation, or automatically performing a task by entering data into multiple webpages. In some embodiments, this is achieved by use of techniques such as natural language processing (NLP) and machine learning to enable the automation of data aggregation and other tasks involving websites without the use of pre-programmed scripts.
1. A method of automating performance of a task for a user, comprising:
receiving from a user an identification of a website and a task they want to have automated;
navigating to a first webpage associated with the website;
extracting one or more intent model features from the first webpage, the one or more intent model features including visual and textual features;
accessing a trained intent model and providing the extracted intent model features from the first webpage as an input to the trained intent model, wherein the trained intent model operates to generate a prediction of one or more actions associated with the webpage and a corresponding confidence level for each of the one or more actions;
extracting one or more target model features from the first webpage, the one or more target model features including both visual and textual features;
accessing a trained target model and providing the extracted target model features from the first webpage and the generated prediction of one of the one or more actions associated with the webpage as an input to the trained target model, wherein for each of the one or more actions associated with the webpage, the trained target model operates to generate a prediction of one or more elements of the webpage that will cause the action to be performed and a corresponding confidence level for each of the one or more elements;
executing one of the one or more actions associated with the webpage by interacting with a highest ranked element of the webpage for causing the one of the one or more actions, wherein executing the one of the one or more actions results in either navigating to a second webpage or performing the task; and
if the task has not been performed, then repeating one or more of the preceding steps on the second webpage and on any subsequent webpages until the task is performed.
2. The method of claim 1 , wherein the trained intent model comprises a multiple component model, and further, wherein the trained intent model utilizes one or more of a natural language processing technology and a machine learning technology to predict the action associated with the webpage.
3. The method of claim 1 , wherein the method further comprises receiving a set of credentials from the user and using the received credentials to login to a user account accessible from the website.
4. The method of claim 1 , wherein the extracted intent model visual features include an image of the webpage, and the textual features include one or more of text contained in a document object model or file describing elements of the webpage and text visible on the webpage.
5. The method of claim 1 , wherein the extracted target model visual features include an image of an element on the webpage and the textual features include one or more of text contained in a document object model or file describing elements of the webpage and text visible on the webpage.
6. The method of claim 1 , wherein the trained target model comprises a multiple component model, and further, wherein the trained target model utilizes one or more of a natural language processing technology and a machine learning technology to predict the element of the webpage that will cause the action associated with the webpage to be performed.
7. The method of claim 1 , wherein the task is one of logging into an account, accessing account data, entering data, submitting a form, or providing payment for a product or service.
8. The method of claim 1 , further comprising storing a record of which element on the webpage has been interacted with and which action has been executed for the webpage, and further, determining if executing the most recently executed action associated with the webpage has increased or decreased the likelihood of performing the task.
9. A non-transitory computer readable medium containing a set of computer-executable instructions which when executed by a processor or processors cause the processor or processors to automate performance of a task for a user by:
receiving from a user an identification of a website and a task they want to have automated;
navigating to a first webpage associated with the website;
extracting one or more intent model features from the first webpage, the one or more intent model features including visual and textual features;
accessing a trained intent model and providing the extracted intent model features from the first webpage as an input to the trained intent model, wherein the trained intent model operates to generate a prediction of one or more actions associated with the webpage and a corresponding confidence level for each of the one or more actions;
extracting one or more target model features from the first webpage, the one or more target model features including both visual and textual features;
accessing a trained target model and providing the extracted target model features from the first webpage and the generated prediction of one of the one or more actions associated with the webpage as an input to the trained target model, wherein for each of the one or more actions associated with the webpage, the trained target model operates to generate a prediction of one or more elements of the webpage that will cause the action to be performed and a corresponding confidence level for each of the one or more elements;
executing one of the one or more actions associated with the webpage by interacting with a highest ranked element of the webpage for causing the one of the one or more actions, wherein executing the one of the one or more actions results in either navigating to a second webpage or performing the task; and
if the task has not been performed, then repeating one or more of the preceding steps on the second webpage and on any subsequent webpages until the task is performed.
10. The non-transitory computer readable medium of claim 9 , further comprising instructions which cause the processor or processors to receive a set of credentials from the user and use the received credentials to login to a user account accessible from the website.
11. The non-transitory computer readable medium of claim 9 , wherein the task is one of logging into an account, accessing account data, entering data, submitting a form, or providing payment for a product or service.
12. The non-transitory computer readable medium of claim 9 , further comprising instructions which cause the processor or processors to store a record of which element on the webpage has been interacted with and which action has been executed for the webpage, and further, determine if executing the most recently executed action associated with the webpage has increased or decreased the likelihood of performing the task.
13. A system for automating a task for a user, comprising:
a set of computer-executable instructions stored in a memory; and
a processor or processors configured to execute the set of instructions, wherein when executed, the instructions cause the processor or processors to perform a set of operations comprising
receiving from a user an identification of a website and a task they want to have automated;
navigating to a first webpage associated with the website;
extracting one or more intent model features from the first webpage, the one or more intent model features including visual and textual features;
accessing a trained intent model and providing the extracted intent model features from the first webpage as an input to the trained intent model, wherein the trained intent model operates to generate a prediction of one or more actions associated with the webpage and a corresponding confidence level for each of the one or more actions;
extracting one or more target model features from the first webpage, the one or more target model features including both visual and textual features;
accessing a trained target model and providing the extracted target model features from the first webpage and the generated prediction of one of the one or more actions associated with the webpage as an input to the trained target model, wherein for each of the one or more actions associated with the webpage, the trained target model operates to generate a prediction of one or more elements of the webpage that will cause the action to be performed and a corresponding confidence level for each of the one or more elements;
executing one of the one or more actions associated with the webpage by interacting with a highest ranked element of the webpage for causing the one of the one or more actions, wherein executing the one of the one or more actions results in either navigating to a second webpage or performing the task; and
if the task has not been performed, then repeating one or more of the preceding steps on the second webpage and on any subsequent webpages until the task is performed.
14. The system of claim 13 , wherein the instructions cause the processor or processors to receive a set of credentials from the user and use the received credentials to login to a user account accessible from the website.
15. The system of claim 13 , wherein the task is one of logging into an account, accessing account data, entering data, submitting a form, or providing payment for a product or service.
16. The system of claim 13 , wherein the instructions cause the processor or processors to store a record of which element on the webpage has been interacted with and which action has been executed for the webpage, and further, determine if executing the most recently executed action associated with the webpage has increased or decreased the likelihood of performing the task.