IP Library › Granted Patent US 10,817,314
Granted Patent B1
US 10,817,314 · App. 16/590,063 · Granted Oct 27, 2020

Augmented shareable video files for robotic process automation

Inventors: Dhurai Ganesan (Chennai, IN); Sankar Chandrasekaran (Chennai, IN); Harsh Vinayak (Gurgaon, IN); Tanvir Khan (Allen, TX)
Assignee: NTT DATA Services, LLC
G06F9/451G06F40/14
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,817,314
App. No.
16/590,063
Filed
Oct 1, 2019
Granted
Oct 27, 2020
Kind
B1
Art Unit
2174
USPC
715/704
Abstract

In one general aspect, in an embodiment, a method of performance-enhanced machine-learning model creation is performed by a computer system. The method includes receiving a command to record user interface (UI) activity in a computing environment. The method further includes, responsive to the command: receiving video frames of a live screen output of the computing environment; detecting UI events in the computing environment in relation to the video frames of the live screen output; and determining target applications for the UI events, wherein the target applications are executing in the computing environment. The method also includes generating UI metadata comprising information identifying the UI events and the target applications in relation to the video frames. In addition, the method includes sequentially encoding, in a video file, the video frames together with information sufficient to derive the UI metadata.

Claims (53)

1. A method of robotic process automation, the method comprising, by a computer system:

receiving a command to record user interface (UI) activity in a computing environment;

responsive to the command:

receiving video frames of a live screen output of the computing environment;

detecting UI events in the computing environment in relation to the video frames of the live screen output; and

determining target applications for the UI events, wherein the target applications are executing in the computing environment;

generating UI metadata comprising information identifying the UI events and the target applications in relation to the video frames; and

sequentially encoding, in a video file, the video frames together with information sufficient to derive the UI metadata.

2. The method of claim 1 , wherein the generating comprises generating structured UI metadata in a markup language.

3. The method of claim 1 , comprising publishing the video file to a user.

4. The method of claim 3 , comprising overlaying supplemental content on the encoded video frames of the video file prior to the publishing.

5. The method of claim 1 , wherein the sequentially encoding comprises encrypting, in the video file, the information sufficient to derive the UI metadata.

6. The method of claim 1 , wherein the information sufficient to derive the UI metadata comprises a representation of the UI metadata.

7. The method of claim 1 , wherein:

the encoded video frames of the video file depicts a user-executed process in the computing environment; and

the encoded information sufficient to derive the UI metadata comprises an instruction set for a bot.

8. The method of claim 1 , wherein the UI events correspond to input from an input device selected form the group consisting of keyboard, mouse, and touchscreen.

9. The method of claim 1 , comprising:

receiving the video file;

decoding, from the video file, the information sufficient to derive the UI metadata;

generating the UI metadata from the decoded information; and

creating a bot using the UI metadata.

10. The method of claim 9 comprising, prior to the creating, determining a complexity of an instruction set for the bot.

11. The method of claim 10 , wherein the creating the bot is performed automatically without user configuration in response to the determined complexity satisfying criteria for automatic bot creation.

12. The method of claim 10 , comprising facilitating user configuration of the instruction set in response to the determined complexity not satisfying criteria for automatic bot creation.

13. The method of claim 9 comprising, responsive to the automatically creating, automatically deploying the bot on a robotic process automation platform.

14. The method of claim 11 , wherein the deployed bot executes a user-executed process depicted in the video file as a robotic process.

15. A computer system comprising a processor and memory, wherein the processor and the memory in combination are operable to implement a method comprising:

receiving a command to record user interface (UI) activity in a computing environment;

responsive to the command:

receiving video frames of a live screen output of the computing environment;

detecting UI events in the computing environment in relation to the video frames of the live screen output; and

determining target applications for the UI events, wherein the target applications are executing in the computing environment;

generating UI metadata comprising information identifying the UI events and the target applications in relation to the video frames; and

sequentially encoding, in a video file, the video frames together with information sufficient to derive the UI metadata.

16. The computer system of claim 15 , wherein the generating comprises generating structured UI metadata in a markup language.

17. The computer system of claim 15 , wherein the sequentially encoding comprises encrypting, in the video file, the information sufficient to derive the UI metadata.

18. The computer system of claim 15 , the method comprising:

receiving the video file;

decoding, from the video file, the information sufficient to derive the UI metadata;

generating the UI metadata from the decoded information; and

creating a bot using the UI metadata.

19. The computer system of claim 15 , the method comprising:

prior to the creating, determining a complexity of an instruction set for the bot; and

wherein the creating the bot is performed automatically without user configuration in response to the determined complexity satisfying criteria for automatic bot creation.

20. A computer-program product comprising a non-transitory computer-usable medium having computer-readable program code embodied therein, the computer-readable program code adapted to be executed to implement a method comprising:

receiving a command to record user interface (UI) activity in a computing environment;

responsive to the command:

receiving video frames of a live screen output of the computing environment;

detecting UI events in the computing environment in relation to the video frames of the live screen output; and

determining target applications for the UI events, wherein the target applications are executing in the computing environment;

generating UI metadata comprising information identifying the UI events and the target applications in relation to the video frames; and

sequentially encoding, in a video file, the video frames together with information sufficient to derive the UI metadata.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jan 13, 2020
From: GANESAN, DHURAI; CHANDRASEKARAN, SANKAR; VINAYAK, HARSH; KHAN, TANVIR
To: NTT DATA SERVICES, LLC
Reel/Frame 051494/0711 →
Cited By (5)
US 12,190,143 US 12,360,784 US 12,379,945 US 12,596,343 US 12,638,819