IP Library Granted Patent US 11,340,920
Granted Patent B2
US 11,340,920 · App. 15/423,312 · Granted May 24, 2022

Device navigational maps for connected devices

Inventors: Florin Darie (Middletown, NJ); Deepak Gonsalves (Bridgewater, NJ); Pooja Chengappa (Bridgewater, NJ)
Assignee: AetherPal Inc.
G06F9/453G06F9/4451G06F11/302G06F16/904G06F9/455G06F11/3006G06F11/3684G06F11/3696
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,340,920
App. No.
15/423,312
Granted
May 24, 2022
Kind
B2
Abstract

Described herein is a method for creating and utilizing device navigational maps. The device navigational maps are modeled using directed graphs that contain detailed information of all components on the connected devices. An application systematically scans through the entire connected device and builds a detailed data model of the connected device, which is used as the navigational map.

Claims (106)

1. A method for creating and using a navigational map for a connected device comprising:

scanning a remote connected device over a network via a remote control connection to generate a set of vertices and edges, wherein each vertex represents a screen displayed by the remote connected device and each edge represents an action to move between a respective pair of screens represented by a respective pair of vertices;

building a navigational map that comprises the set of vertices and edges;

providing the navigational map to a navigational map server over the network;

receiving an inquiry from a user associated with the remote connected device, the inquiry comprising a keyword;

providing a list of screens including the keyword to the remote connected device responsive to the inquiry, individual screens of the list of screens comprising a screen area including the keyword;

receiving from the remote connected device an indication of a destination screen from the list of screens;

computing a path between a start screen and the destination screen using the navigational map; and

sending a series of commands to the remote connected device, wherein execution of the series of commands causes the remote connected device to navigate the path.

2. The method of claim 1 , further comprising:

monitoring a current screen against the navigational map; and

matching the current screen with the navigational map.

3. The method of claim 1 , further comprising:

modifying the navigational map with screens not present in the navigational map; and

providing the modified navigational map to the navigational map server over the network.

4. The method of claim 1 , further comprising:

setting a depth level for a predetermined set of remote connected device components, wherein scanning is completed upon reaching the depth level.

5. The method of claim 1 , further comprising:

creating a queue of unvisited vertices, wherein the queue is updating during scanning of the remote connected device; and

for each vertex in the queue of unvisited vertices:

determining other unvisited vertices that can be reached;

creating a query vertex for each unvisited vertex;

determining if the query vertex already exists in the navigational map;

creating a new edge if the query vertex does not exist; and

decrementing the queue when all vertices for a depth level is complete.

6. The method of claim 5 , further comprising checking all links and scrollable elements to determine an existence of the other unvisited vertices.

7. The method of claim 1 , wherein scanning comprises:

sending a command over the network via the remote control connection to the remote connected device to switch to a home screen;

obtaining a first screen image of the home screen and a first screen layout of the home screen;

creating a first vertex that comprises the first screen image of the home screen and the first screen layout of the home screen;

adding the first vertex to the set of vertices and edges;

identifying a second screen reachable from the home screen in response to an interaction with a user interface element rendered within the home screen;

obtaining a second screen image of the second screen and a second screen layout of the second screen;

creating a second vertex that comprises the second screen image and the second screen layout;

adding the second vertex to the set of vertices and edges;

creating an edge that connects the first vertex with the second vertex, the edge comprising the interaction with the user interface element rendered within the home screen; and

adding the edge to the set of vertices and edges.

8. A system for creating and using a navigational map for a connected device, comprising:

a computing device comprising a processor and a memory; and

a navigation map builder stored in the memory that, when executed by the processor, causes the computing device to at least:

scan a remote connected device over a network via a remote control connection to generate a set of vertices and edges, wherein each vertex represents a screen displayed by the remote connected device and each edge represents an action to move between a respective pair of screens represented by a respective pair of vertices;

build a navigational map that comprises the set of vertices and edges; and

provide the navigation map to a navigational map server over the network; and

a navigation map service stored in the memory, wherein when the navigation map service receives an inquiry comprising a keyword from a user associated with the remote connected device, causes the computing device to at least:

provide a list of screens including the keyword to the remote connected device responsive to the inquiry, individual screens of the list of screens comprising a screen area including the keyword;

receive from the remote connected device a selection of a destination screen from the list of screens;

compute a path between a start screen and the destination screen using the navigational map; and

send a series of commands to the remote connected device to navigate the path.

9. The system of claim 8 , wherein the navigation map service further comprises a screen monitor, the screen monitor monitoring a current screen against the navigational map and matches the current screen with the navigational map.

10. The system of claim 8 , wherein the navigation map service further comprises a navigational map extender, the navigational map extender modifying the navigational map with screens not present in the navigational map and providing the modified navigational map to the navigational map server over the network.

11. The system of claim 8 , wherein the navigation map builder sets a depth level for a predetermined set of remote connected device components, and wherein scanning is completed upon reaching a designated depth level.

12. The system of claim 8 , wherein the navigation map builder:

creates a queue of unvisited vertices, wherein the queue is updating during scanning of the remote connected device; and

for each vertex:

determines other unvisited vertices that can be reached;

creates a query vertex for each unvisited vertex;

determines if the query vertex already exists in the navigational map;

creates a new edge if the query vertex does not exist; and

decrements the queue when all vertices for a depth level is complete.

13. The system of claim 12 , wherein all links and scrollable elements are checked to determine existence of the other unvisited vertices.

14. The system of claim 8 , wherein when the navigation map builder causes the computing device to scan the remote connected device, the navigation map builder further causes the computing device to at least:

send a command over the network via the remote control connection to the remote connected device to switch to a home screen;

obtain a first screen image of the home screen and a first screen layout of the home screen;

create a first vertex that comprises the first screen image of the home screen and the first screen layout of the home screen;

add the first vertex to the set of vertices and edges;

identify a second screen reachable from the home screen in response to an interaction with a user interface element rendered within the home screen;

obtain a second screen image of the second screen and a second screen layout of the second screen;

create a second vertex that comprises the second screen image and the second screen layout;

add the second vertex to the set of vertices and edges;

create an edge that connects the first vertex with the second vertex, the edge comprising the interaction with the user interface element rendered within the home screen; and

add the edge to the set of vertices and edges.

15. A non-transitory, computer-readable medium embodying a program that, when executed in at least one computing device, causes the at least one computing device to at least:

scan a remote connected device over a network via a remote control connection to generate a set of vertices and edges, wherein each vertex represents a screen displayed by the remote connected device and each edge represents an action to move between a respective pair of screens represented by a respective pair of vertices;

build a navigational map that comprises the set of vertices and edges;

provide the navigational map to a navigational map server over the network;

receive an inquiry from a user associated with the remote connected device, the inquiry comprising a keyword;

provide a list of screens including the keyword to the remote connected device responsive to the inquiry, individual screens of the list of screens comprising a screen area including the keyword;

receive from the remote connected device an indication of a destination screen from the list of screens;

compute a path between a start screen and the destination screen using the navigational map; and

send a series of commands to the remote connected device, wherein execution of the series of commands causes the remote connected device to navigate the path.

16. The non-transitory, computer-readable medium of claim 15 , wherein the program, when executed by the at least one computing device, further causes the at least one computing device to at least:

monitor a current screen against the navigational map; and

match the current screen with the navigational map.

17. The non-transitory, computer-readable medium of claim 15 , wherein the program, when executed by the at least one computing device, further causes the at least one computing device to at least:

create a queue of unvisited vertices, wherein the queue is updating during scanning of the remote connected device; and

for each vertex:

determine other unvisited vertices that can be reached;

create a query vertex for each unvisited vertex;

determine if the query vertex already exists in the navigational map;

create a new edge if the query vertex does not exist; and

decrement the queue when all vertices for a depth level is complete.

18. The non-transitory, computer-readable medium of claim 15 , wherein the program, when executed by the at least one computing device, further causes the at least one computing device to at least:

modify the navigational map with screens not present in the navigational map; and

provide the modified navigational map to the navigational map server over the network.

19. The non-transitory, computer-readable medium of claim 15 , wherein the program, when executed by the at least one computing device, further causes the at least one computing device to at least set a depth level for a predetermined set of connected device components, wherein scanning is completed upon reaching the depth level.

20. The non-transitory, computer-readable medium of claim 19 , wherein when the program causes the at least one computing device to scan the remote connected device over the network via the remote control connection, the program further causes the at least one computing device to at least:

send a command over the network via the remote control connection to the remote connected device to switch to a home screen;

obtain a first screen image of the home screen and a first screen layout of the home screen;

create a first vertex that comprises the first screen image of the home screen and the first screen layout of the home screen;

add the first vertex to the set of vertices and edges;

identify a second screen reachable from the home screen in response to an interaction with a user interface element rendered within the home screen;

obtain a second screen image of the second screen and a second screen layout of the second screen;

create a second vertex that comprises the second screen image and the second screen layout;

add the second vertex to the set of vertices and edges;

create an edge that connects the first vertex with the second vertex, the edge comprising the interaction with the user interface element rendered within the home screen; and

add the edge to the set of vertices and edges.

Assignments (5)
PATENT ASSIGNMENT Recorded Aug 5, 2024
From: VMWARE LLC
To: OMNISSA, LLC
Reel/Frame 068327/0365 →
SECURITY INTEREST Recorded Jul 3, 2024
From: OMNISSA, LLC
To: UBS AG, STAMFORD BRANCH
Reel/Frame 068118/0004 →
CHANGE OF NAME Recorded Apr 15, 2024
From: VMWARE, INC.
To: VMWARE LLC
Reel/Frame 067102/0395 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jun 23, 2021
From: AETHERPAL INC.
To: VMWARE, INC.
Reel/Frame 056630/0792 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Apr 16, 2019
From: DARIE, FLORIN; GONSALVES, DEEPAK; CHENGAPPA, POOJA
To: AETHERPAL INC.
Reel/Frame 048917/0463 →