Interactive website automation for health care web portals with random content
A system and method for controlling a headless browser automation (an automation that does not render web pages or involve direct user input), where the web portals accessed by the browser automation employ a rule system for generating web content and the rules can result in the dynamic presentation of new response elements that could not be a priori determined from existing content already encountered, or the history of interaction with the web portal. The control system has the capability to pause the headless browser automation in place, collect the new information being requested by the web portal, present the requested new information to a remote user, capture the information from the remote user, feed the information collected to the portal automation, and restart the automation from the point that it left off.
1. A computer-implemented method for controlling a headless browser automation comprising:
scraping, by one or more processors, a plurality of web portals each configured for processing insurance coverage requests;
receiving, by the one or more processors, a dynamic content containing data in a presentation specific form from one or more of the plurality of web portals;
pausing, by the one or more processors, scraping of the plurality of web portals, based on determining that the dynamic content requires engaging a user before the scraping of the one or more of the plurality of web portals can continue;
extracting from the dynamic content, by the one or more processors, context information, a query for a desired response and a data set for utilization management comprising insurer decisions based on insurance plan benefits;
converting, by the one or more processors, the context information, the query for the desired response, and the extracted data set into a modal data structure comprising presentation agnostic converted data;
presenting, by the one or more processors, the converted data to the user via a human user rendering and response collection system;
receiving, by the one or more processors, data responsive to the query for the desired response from the user; and
resuming, by the one or more processors, scraping of the plurality of web portals using the received data responsive to the query for desired response.
2. The method of claim 1 , wherein the human user rendering and response collection system includes a man-machine interface selected from a group consisting of a touchscreen, a voice response system, a user interface designed for use by a human with disabilities, a brain-computer interface, or a combination thereof.
3. The method of claim 1 , wherein the converted data is presented to the user using a representational state transfer (REST) application programming interface (API), wherein the data responsive to the query for desired response is received from the user using the REST API, or both.
4. The method of claim 1 , wherein the data in the received dynamic content is converted into the modal data structure by a sub function configured to support a plurality of dynamic content response use cases.
5. The method of claim 4 , wherein the plurality of dynamic content response use cases include a request to submit free text response, a request to enter multi-factor authentication code, and a request to answer multiple choice question.
6. The method of claim 4 , further comprising extracting a question dictionary from the dynamic content, and wherein the sub function produces the modal data structure from the question dictionary.
7. The method of claim 6 , wherein the modal data structure produced from the question dictionary is specific to each of the plurality of dynamic content response use cases.
8. The method of claim 6 , further comprising converting the modal data structure into a language-independent data format before the presenting.
9. The method of claim 6 , wherein the data structure is in a Python dictionary format.
10. The method of claim 8 , wherein the language-independent data format is JavaScript Object Notation.
11. The method of claim 1 , wherein the presentation specific form is specific to each of the plurality of web portals.
12. The method of claim 1 , wherein the data set for utilization management includes one or more prior authorization requests.
13. An apparatus for controlling a headless browser automation comprising at least one processor coupled to a memory, the memory holding program instructions that when executed by the at least one processor cause the apparatus to perform:
scraping a plurality of web portals each configured for processing insurance coverage requests;
receiving a dynamic content containing data in a presentation specific form from one or more of the plurality of web portals;
pausing scraping of the plurality of web portals, based on determining that the dynamic content requires engaging a user before the scraping of the one or more of the plurality of web portals can continue;
extracting from the dynamic content context information, a query for a desired response, and a data set for utilization management comprising insurer decisions based on insurance plan benefits;
converting the context information, the query for the desired response, and the data set into a modal data structure comprising presentation agnostic converted data;
presenting the converted data to a user via a human user rendering and response collection system;
receiving data responsive to the query for the desired response from the user; and
resuming scraping of the plurality of web portals using the data responsive to the query for the received desired response.
14. The apparatus of claim 13 , wherein the human user rendering and response collection system includes a man-machine interface selected from a group consisting of a touchscreen, a voice response system, a user interface designed for use by a human with disabilities, a brain-computer interface, or a combination thereof.
15. The apparatus of claim 13 , wherein the converted data is presented to the user using a representational state transfer (REST) application programming interface (API), wherein the data responsive to the query for the desired response is received from the user using the REST API, or both.
16. The apparatus of claim 13 , wherein the data in the received dynamic content is converted into the modal data structure by a sub function configured to support a plurality of dynamic content response use cases and executed by the at least one processor.
17. The apparatus of claim 16 , wherein the plurality of dynamic content response use cases include a request to submit free text response, a request to enter multi-factor authentication code, and a request to answer multiple choice question.
18. The apparatus of claim 16 , further comprising extracting a question dictionary from the dynamic content, and wherein the sub function produces the modal data structure from the question dictionary.
19. The apparatus of claim 18 , wherein the modal data structure produced from of the question dictionary is specific to each of the plurality of dynamic content response use cases.
20. The apparatus of claim 18 , further comprising converting the modal data structure into a language-independent data format before the presenting.
21. The apparatus of claim 13 , wherein the modal data structure is in a Python dictionary format.
22. The apparatus of claim 20 , wherein the language-independent data format is JavaScript Object Notation.
23. The apparatus of claim 13 , wherein the presentation specific form is specific to each of the plurality of web portals.
24. The apparatus of claim 13 , wherein the data set for utilization management includes one or more prior authorization requests.