Granted Patent B2

US 12,619,458 · App. 17/726,053 · Granted May 5, 2026

Inventors: Micheal Dunn (Redmond, WA); Nabeel Shahzad (Sammamish, WA); Sri Raghu Malireddi (Bellevue, WA); Jonathan W. Lin (Seattle, WA); Olutayo Falase (Seattle, WA); Deyuan Ke (Redmond, WA); Joann T. Lee (Redmond, WA); Shrey Shah (Redmond, WA)

Assignee: Microsoft Technology Licensing, LLC

G06F9/4831G06F40/205G06F40/35

Abstract

Claims (79)

1 . A system comprising:

at least one processor; and

memory storing instructions that, when executed by the at least one processor, causes the system to perform a set of operations, the set of operations comprising:

initiating an automated workflow, the automated workflow comprising a plurality of workflow states, the initiating comprising loading an indication of a workflow state into the memory to facilitate performing the operations to automate the workflow;

detecting a change from a first workflow state to a second workflow state;

extracting a document object model (DOM) of the second workflow state;

analyzing the DOM to detect a first DOM element comprising markup content;

programmatically detecting, based on the markup content and one or more attributes of the first DOM element, a workflow interruption requiring a user input;

classifying, based on the one or more attributes of the first DOM element, a type of workflow interruption by comparing the one or more attributes to DOM attributes associated with known workflow interruptions;

modifying execution of the automated workflow based on the classified type of workflow interruption, including pausing the automated workflow and storing an indication of the paused workflow state in the memory;

causing presentation of a representation of the second workflow state to a user;

receiving an indication that the user input has been received; and

based on receiving the indication, accessing from the memory the indication of the paused workflow state and resuming the automated workflow.

2 . The system of claim 1 , wherein the automated workflow is associated with a set of workflow instructions, the workflow instructions comprising a series of actions for proceeding through the plurality of workflow states.

3 . The system of claim 1 , wherein the workflow interruption comprises a CAPTCHA request.

4 . The system of claim 3 , wherein the user input has been received when the CAPTCHA request has been completed.

5 . The system of claim 1 , wherein

the first DOM element comprises an iframe, and the markup content includes a keyword.

6 . The system of claim 5 , wherein the keyword indicates a “CAPTCHA” interruption.

7 . The system of claim 1 ,

wherein the known workflow interruptions include a prior workflow interruption;

wherein the first DOM element is associated with a visual feature of a user interface presented during the second workflow state;

wherein classifying the type of workflow interruption further comprises:

identifying the visual feature; and

determining a degree of similarity between the visual feature and a visual feature associated with the prior workflow interruption;

wherein the degree of similarity is represented by a visual feature score, and

wherein the type of workflow interruption is determined based on the visual feature score exceeding a threshold.

8 . The system of claim 7 , wherein the set of operations further comprises:

determining pixel intensities of the first DOM element or a second DOM element of the DOM of the second workflow state; and

generating a pixel intensity score indicating a degree of similarity between the pixel intensities of the first or second DOM elements and pixel intensities associated with the prior workflow interruption; and

classifying the type of workflow interruption based on a combination of the visual feature score and the pixel intensity score.

9 . The system of claim 8 , wherein the pixel intensity score further indicates a first workflow-interruption type to which the pixel intensities of the second workflow state are the most similar, and wherein the visual features score further indicates a second workflow-interruption type to which the visual feature of the second workflow state is the most similar.

10 . The system of claim 9 , wherein determining that the second workflow state includes a workflow interruption comprises:

generating a computer-vision score based on the pixel-intensity score and the visual-feature score;

if the first workflow-interruption type and the second workflow-interruption type are the same, comparing the computer-vision score to a first threshold and determining that the computer-vision score exceeds the first threshold; and

if the first workflow-interruption type and the second workflow-interruption type are not the same, comparing the computer-vision score to a second threshold and determining that the computer-vision score exceeds the first threshold, wherein the second threshold is higher than the first threshold.

11 . The system of claim 1 ,

wherein classifying the type of workflow interruption comprises comparing the one or more attributes of the first DOM element to DOM attributes associated with known workflow interruptions stored in interruption detection data.

12 . A computer-implemented method for detecting a workflow interruption, the method comprising:

initiating an automated workflow, the automated workflow comprising a plurality of workflow states;

detecting a change from a first workflow state to a second workflow state;

extracting a document object model (DOM) of the second workflow state;

analyzing the DOM to detect a DOM element comprising markup content;

programmatically detecting, based on the markup content and one or more attributes of the DOM element, a workflow interruption requiring a user input;

classifying, based on the one or more attributes of the DOM element, a type of workflow interruption by comparing the one or more attributes to DOM attributes associated with known workflow interruptions;

modifying execution of the automated workflow based on the classified type of workflow interruption, including pausing the automated workflow;

causing presentation of the second workflow state to a user;

receiving an indication that the user input has been received; and

based on receiving the indication, resuming the automated workflow.

13 . The computer-implemented method of claim 12 , wherein the workflow interruption comprises a CAPTCHA request.

14 . The computer-implemented method of claim 13 , wherein examining the attribute of the second workflow state comprises:

determining, based on the indication of the DOM stored in the database, that the second workflow state comprises an iframe; and

identifying the keyword in a markup of the iframe, wherein the keyword is “CAPTCHA”.

15 . The computer-implemented method of claim 13 , wherein examining the attribute of the second workflow state comprises:

identifying a visual feature of a user interface presented during the second workflow state; and

determining a degree of similarity between the visual feature of the user interface associated with the second workflow state and a visual feature associated with a prior workflow interruption.

16 . The computer-implemented method of claim 13 , wherein examining the attribute of the second workflow state comprises:

determining pixel intensities of the user interface associated with the second workflow state; and

determining a degree of similarity between the pixel intensities of the user interface and pixel intensities of a prior workflow interruption.

17 . The computer-implemented method of claim 14 , wherein examining the attribute of the second workflow state comprises further comprises:

identifying a visual feature of a user interface presented during the second workflow state; and

determining a degree of similarity between the visual feature of the user interface associated with the second workflow state and a visual feature associated with a prior workflow interruption.

18 . The computer-implemented method of claim 17 , wherein determining the degree of similarity comprises generating a visual feature score.

19 . The computer-implemented method of claim 18 , wherein determining, based on the examining, that the second workflow state includes a workflow interruption comprises comparing the visual feature score to a visual feature threshold.

20 . A non-transitory computer-storage medium encoding computer-executable instructions that, when executed by at least one processor, perform a method comprising:

initiating an automated workflow, the automated workflow comprising a plurality of workflow states;

detecting a change from a first workflow state to a second workflow state;

identifying a visual feature of a user interface presented during the second workflow state;

storing an indication of the visual feature in a memory;

executing a computer vision program to process the indication of the visual feature by:

accessing from the memory an indication of and a prior visual feature associated with a prior workflow interruption;

generating, based on the indication of the visual feature stored in the memory, a visual feature score indicating a degree of similarity between the visual feature of the user interface associated with the second workflow state and the prior visual feature associated with the prior workflow interruption;

determining pixel intensities of the visual feature of the user interface associated with the second workflow state;

accessing, from the memory, pixel intensities of the prior visual feature associated with the prior workflow interruption; and

generating a pixel intensity score indicating a degree of similarity between the pixel intensities of the visual feature of the user interface associated with the second workflow state and pixel intensities of the prior visual feature associated with the prior workflow interruption;

determining, based on the visual feature score and the pixel intensity score, that the second workflow state includes a workflow interruption;

pausing the automated workflow;

receiving an indication that the user input has been received; and

based on receiving the indication, resuming the automated workflow.

Assignments (1)

Continuity (1)

References Cited (35)