Granted Patent B2

US 11,822,873 · App. 17/120,412 · Granted Nov 21, 2023

Inventor: Manbinder Pal Singh (Coral Springs, FL)

G06F40/166G06F40/103G10L15/22G10L15/26

Abstract

Claims (95)

1. A method, comprising:

causing, by a computing system, first output content of an application to be displayed, wherein the first output content includes at least a first user interface element, and the application is configured to perform a first action in response to selection of the first user interface element;

receiving, by the computing system and while the first output content is being displayed, a first user input indicating that a first step in a sequence of steps is to be recorded for inclusion in a document relating to the application;

in response to the first user input, precluding the application from performing the first action in response to selection of the first user interface element;

receiving, by the computing system and while the application is precluded from performing the first action in response to selection of the first user interface element, a second user input selecting the first user interface element;

storing, by the computing system and based at least in part on receipt of the second user input while the application was precluded from performing the first action in response to selection of the first user interface element, a first record for the first step, the first record including at least a first sequence identifier, first data representing the first user interface element, the first sequence identifier indicating a relative position of the first step within the sequence of steps;

after receiving the second user input, enabling the application to perform the first action in response to selection of the first user interface element; and

generating, by the computing system, the document at least in part by using the first data to generate a first visual portion for the first step, using the first sequence identifier to determine a position of the first visual portion, in the document, relative to one or more other visual portions representing one or more other steps in the sequence of steps.

2. The method of claim 1 , further comprising:

after enabling the application to perform the first action in response to selection of the first user interface element, receiving, by the computing system, a third user input selecting the first user interface element;

in response to receipt of the third user input, causing the application to perform the first action;

based at least in part on performance of the first action by the application, causing the computing system to display second output content, wherein the second output content includes at least a second user interface element, and the application is configured to perform a second action in response to selection of the second user interface element;

receiving, by the computing system and while the second output content is being displayed, a fourth user input indicating that a second step in a sequence of steps is to be recorded for inclusion in the document;

in response to the fourth user input, precluding the application from performing the second action in response to selection of the second user interface element;

receiving, by the computing system, a fifth user input selecting the;

storing, by the computing system and based at least in part on receipt of the fifth user input while the application was precluded from performing the second action in response to selection of the second user interface element, a second record for the second step, the second record including at least a second sequence identifier and second data representing the second user interface element, the second sequence identifier indicating a relative position of the second step within the sequence of steps; and

after receiving the fifth user input, enabling the application to perform the second action in response to selection of the second user interface element;

wherein generating, by the computing system, the document further comprises generating the document using the second data to generate a second visual portion for the second step, and using the second sequence identifier to determine a position of the second visual portion, in the document, relative to at least the first visual portion.

3. The method of claim 1 , further comprising:

receiving a third user input indicating that the first user interface element or a second user interface element included in the first output content is to be marked up for the first step;

wherein the first record further includes second data representing a markup element for the first user interface element or the second user interface element, and

wherein the first visual portion is generated using the first data and the second data, the first visual portion representing the markup element surrounding the first user interface element or the second user interface element.

4. The method of claim 1 , wherein:

the application is accessed using a web browser;

the first data includes a representation of a document object model (DOM) element corresponding to the first user interface element; and

the first visual portion includes an image generated using the DOM element.

5. The method of claim 1 , wherein the first data includes a representation of an image of a portion of the first output content within which the first user interface element is displayed.

6. The method of claim 1 , further comprising prior to receiving the first user input:

causing, by the computing system, a document generation application to be launched; and

causing, by the document generation application, a user interface to be displayed in association with the first output content, the user interface including a third user interface element that is selectable to enable a user to provide the first user input to the computing system.

7. The method of claim 1 , further comprising:

receiving, by the computing system, a voice input representing a description for the first step; and

processing, by the computing system, the voice input to determine text data for the first step;

wherein the first record further includes the text data, and

wherein generating the document further comprises generating the document using the text data to generate a transcript portion for the first step, and arranging the transcript portion adjacent the first visual portion.

8. The method of claim 1 , further comprising:

determining, by the computing system, text data representing a description for the first step, the text data being based on a name of the first user interface element and a type of the first user interface element;

wherein the first record further includes the text data, and

wherein generating the document further comprises generating the document using the text data to generate a transcript portion for the first step, and arranging the transcript portion adjacent to the first visual portion in the document.

9. The method of claim 1 , wherein the document is displayed via a web browser, and the method further comprises:

generating, by the computing system, a DOM corresponding to contents of the document, the DOM including at least a first node representing the first visual portion for the first step, and a second node representing a second visual portion for a second step represented in the document.

10. The method of claim 1 , further comprising:

receiving, by the computing system, a third user input enabling an edit mode for the document;

using a DOM corresponding to the document, displaying, by the computing system, the first visual portion for the first step;

receiving, by the computing system, a fourth user input with respect to the first visual portion;

updating, by the computing system, the DOM based on the fourth user input to generate an updated DOM;

generating, by the computing system, an updated document by replacing the first visual portion with an updated visual portion based on the updated DOM; and

storing, by the computing system, the updated document.

11. The method of claim 10 , wherein the fourth user input represents a change in a name of the first user interface element represented in the first visual portion.

12. The method of claim 10 , wherein:

the first visual portion includes a markup element for the first user interface element;

the fourth user input represents deselection of the first user interface element; and

updating the DOM comprises removing, from the DOM, the markup element for the first user interface element.

13. The method of claim 10 , wherein:

the first visual portion includes a first markup element for the first user interface element;

the fourth user input represents selection of a second user interface element represented in the first visual portion; and

updating the DOM comprises:

removing the first markup element from the DOM, and

adding, to the DOM, a second markup element for the second user interface element.

14. A system, comprising:

at least one processor; and

at least one computer-readable medium encoded with instructions which, when executed by the at least one processor, cause the system to:

cause first output content of an application to be displayed, wherein the first output content includes at least a first user interface element, and the application is configured to perform a first action in response to selection of the first user interface element;

receive, while the first output content is being displayed, a first user input indicating that a first step in a sequence of steps is to be recorded for inclusion in a document relating to the application;

in response to the first user input, precluding the application from performing the first action in response to selection of the first user interface element;

receive, while the application is precluded from performing the first action in response to selection of the first user interface element, a second user input selecting the first user interface element;

identify at least a first document object model (DOM) element representing the first user interface element;

store, based at least in part on receipt of the second user input while the application was precluded from performing the first action in response to selection of the first user interface element, a first record for the first step, the first record including the first DOM element, a first sequence identifier, the first sequence identifier indicating a relative position of the first step within the sequence of steps;

after receiving the second user input, enable the application to perform the first action in response to selection of the first user interface element; and

generate the document at least in part by using the first DOM element to generate a first visual portion for the first step, using the first sequence identifier to determine a position of the first visual portion relative to one or more other visual portions representing one or more other steps in the sequence of steps.

15. The system of claim 14 , wherein the at least one computer-readable medium is further encoded with additional instructions which, when executed by the at least one processor, further cause the system to:

receive a third user input indicating that the first user interface element or a second user interface element included in the first output content is to be marked up for the first step;

wherein the first record further includes first data representing a markup element for the first user interface element or the second user interface element, and

wherein the first visual portion is generated using the first DOM element and the first data, the first visual portion representing the markup element surrounding the first user interface element or the second user interface element.

16. The system of claim 14 , wherein the at least one computer-readable medium is further encoded with additional instructions which, when executed by the at least one processor, further cause the system to, prior to receiving the first user input:

cause a document generation application to be launched; and

cause, by the document generation application, a user interface to be displayed in association with the first output content, the user interface including a third user interface element that is selectable to enable a user to provide the first user input to the system.

17. The system of claim 14 , wherein the at least one computer-readable medium is further encoded with additional instructions which, when executed by the at least one processor, further cause the system to:

receive a voice input representing a description for the first step; and

process the voice input to determine text data for the first step;

wherein the first record further includes the text data, and

wherein generation of the document further includes using the text data to generate a transcript portion for the first step, and arranging the transcript portion adjacent the first visual portion.

18. The system of claim 14 , wherein the at least one computer-readable medium is further encoded with additional instructions which, when executed by the at least one processor, further cause the system to:

determine text data representing a description for the first step, the text data being based on a name of the first user interface element and a type of the first user interface element;

wherein the first record further includes the text data, and

wherein generation of the document further includes using the text data to generate a transcript portion for the first step, and arranging the transcript portion adjacent the first visual portion in the document.

19. The system of claim 14 , wherein the at least one computer-readable medium is further encoded with additional instructions which, when executed by the at least one processor, further cause the system to:

generate a DOM corresponding to contents of the document, the DOM including at least a first node representing the first visual portion for the first step, and a second node representing a second visual portion for a second step represented in the document.

20. The system of claim 19 , wherein the at least one computer-readable medium is further encoded with additional instructions which, when executed by the at least one processor, further cause the system to:

receive a third user input enabling an edit mode for the document;

display, using the DOM, the first visual portion for the first step;

receive a fourth user input with respect to the first visual portion;

update the DOM based on the fourth user input to generate an updated DOM;

generate an updated document by replacing the first visual portion with an updated visual portion based on the updated DOM; and

store the updated document and the updated DOM.

Assignments (9)

Continuity (1)