IP Library Granted Patent US 11,120,639
Granted Patent B1
US 11,120,639 · App. 16/857,741 · Granted Sep 14, 2021

Projecting telemetry data to visualization models

Inventors: Jesse MacDonald Smith (Seattle, WA); Eric Charles Youngstrom (Kirkland, WA); Jeremy Brent Jackson (Bellevue, WA); Nicholas James Pavlovsky (Duvall, WA); Jason John Fierst (Sammamish, WA); Daniel Taylor Rosas (Kirkland, WA); George Oliver Johnston (Redmond, WA); Ted Tai-Yu Chen (Seattle, WA)
Assignee: Microsoft Technology Licensing, LLC
G06T19/20G06T19/006G06T2219/2004G06T2219/2016
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,120,639
App. No.
16/857,741
Granted
Sep 14, 2021
Kind
B1
Abstract

A computer system is provided that includes one or more processors configured to define a virtual model of a workspace that is world-locked to a three-dimensional environment by a pair of anchor points. The one or more processors are configured to adjust a fit of the virtual model to the workspace by adjusting a position of a virtual component of the virtual model relative to the pair of anchor points. The one or more processors are configured to receive telemetry data including position information indicating a location of a telemetry event relative to the pair of anchor points in the workspace, and determine a visualization model based on the virtual model of the workspace, and project telemetry data to the visualization model based on a mapping of a pair of points in the visualization model to the pair of anchor points in the virtual model.

Claims (83)

1. A computer system comprising:

one or more processors configured to:

in a set-up phase:

define a virtual model that is representative of a workspace in a real-world three-dimensional environment, wherein the virtual model includes one or more virtual components that represent corresponding real-world objects in the real-world three-dimensional environment, and wherein the virtual model is world-locked to the real-world three-dimensional environment by a pair of anchor points;

adjust a fit of the virtual model to the workspace by adjusting a position of a virtual component of the virtual model relative to the pair of anchor points;

in a runtime phase:

receive telemetry data from a telemetry device, the telemetry data including position information indicating a location of a telemetry event relative to the pair of anchor points in the workspace;

aggregate the received telemetry data in a datastore;

in a visualization phase:

determine a visualization model based on the virtual model of the workspace;

map a pair of points in the visualization model to the pair of anchor points in the virtual model of the workspace;

project the aggregated telemetry data to the visualization model based on the mapping of the pair of points in the visualization model to the pair of anchor points in the virtual model; and

display the visualization model with the aggregated telemetry data via a display of the computer system.

2. The computer system of claim 1 , wherein a scale of the visualization model differs from the virtual model of the workspace; and

wherein the one or more processors are configured to:

determine a scaling difference between the pair of points in the visualization model and the pair of anchor points of the virtual model of the workspace; and

map the position information of the aggregated telemetry data to the pair of points of the visualization model based on the scaling difference.

3. The computer system of claim 1 , wherein the visualization model is a two-dimensional visualization model; and

wherein the one or more processors are configured to project the virtual model of the workspace and the aggregated telemetry data to the two-dimensional visualization model based on the mapping of the pair of points in the visualization model to the pair of anchor points in the virtual model.

4. The computer system of claim 1 , wherein the visualization model is a three-dimensional augmented reality model; and

wherein the one or more processors are configured to display the three-dimensional visualization model via a near-eye display device of the computer system.

5. The computer system of claim 1 , wherein the telemetry event of the received telemetry data is selected from the group consisting of an incident event, a pick event, a put event, a reroute event, a forklift event, a notification event, a scan event, an error event, and a user event.

6. The computer system of claim 1 , wherein the one or more processors are further configured to:

in the runtime phase:

identify a workflow including one or more telemetry events and a route in the workspace; and

receive telemetry data from the telemetry device for a user performing the workflow.

7. The computer system of claim 6 , wherein the one or more processors are further configured to:

in the runtime phase:

compare the aggregated telemetry data to the identified workflow;

determine one or more exception events indicating a deviation from the identified workflow based on the aggregated telemetry data, the one or more exception events including position information indicating a location of the exception event relative to the pair of anchor points in the workspace; and

in the visualization phase:

project the one or more exception events to the visualization model by mapping the position information of the one or more exception events to the pair of points in the visualization model.

8. The computer system of claim 6 , wherein the one or more processors are further configured to output instructions for the workflow to the user indicating the one or more telemetry events and the route in the workspace.

9. The computer system of claim 1 , wherein to define the virtual model of the workspace, the one or more processors are further configured to:

scan the workspace in the real-world three-dimensional environment via a camera system; and

generate a three-dimensional virtual model of the workspace based on data from the scan, the three-dimensional virtual model including the one or more virtual components that that represent corresponding real-world objects in the workspace, the one or more virtual components having positions relative to the pair of anchor points.

10. The computer system of claim 9 , wherein the one or more processors are further configured to:

receive a user alignment input to align a virtual component with a corresponding real-world object in the workspace; and

adjust a position of the virtual component relative to the pair of anchor points based on the user alignment input.

11. A method comprising:

at one or more processors of a computer system:

defining a virtual model that is representative of a workspace in a real-world three-dimensional environment, wherein the virtual model includes one or more virtual components that represent corresponding real-world objects in the real-world three-dimensional environment, and wherein the virtual model is world-locked to the real-world three-dimensional environment by a pair of anchor points, and wherein a fit of the virtual model is aligned to the workspace by use of an alignment parameter;

receiving telemetry data from a telemetry device, the telemetry data including position information indicating a location of a telemetry event relative to the pair of anchor points in the workspace;

aggregating the received telemetry data in a datastore;

determining a visualization model based on the virtual model of the workspace;

mapping a pair of points in the visualization model to the pair of anchor points in the virtual model of the workspace;

projecting the aggregated telemetry data to the visualization model based on the mapping of the pair of points in the visualization model to the pair of anchor points in the virtual model; and

outputting data for the visualization model with the aggregated telemetry data for display.

12. The method of claim 11 , further comprising:

adjusting the fit of the virtual model to the workspace by adjusting a position of a virtual component of the virtual model relative to the pair of anchor points, and storing a value for the adjusting in the alignment parameter to thereby align the fit of the virtual model.

13. The method of claim 11 , wherein a scale of the visualization model differs from the virtual model of the workspace; and

wherein the method further comprises:

determining a scaling difference between the pair of points in the visualization model and the pair of anchor points of the virtual model of the workspace; and

mapping the position information of the aggregated telemetry data to the pair of points of the visualization model based on the scaling difference.

14. The method of claim 11 , wherein the visualization model is a two-dimensional visualization model; and

wherein the method further comprises projecting the virtual model of the workspace and the aggregated telemetry data to the two-dimensional visualization model based on the mapping of the pair of points in the visualization model to the pair of anchor points in the virtual model.

15. The method of claim 11 , wherein the visualization model is a three-dimensional augmented reality model; and

wherein the method further comprises displaying the three-dimensional visualization model via a near-eye display device of the computer system.

16. The method of claim 11 , further comprising:

identifying a workflow including one or more telemetry events and a route in the workspace; and

receiving telemetry data from the telemetry device for a user performing the workflow.

17. The method of claim 16 , further comprising:

comparing the aggregated telemetry data to the identified workflow;

determining one or more exception events indicating a deviation from the identified workflow based on the aggregated telemetry data, the one or more exception events including position information indicating a location of the exception event relative to the pair of anchor points in the workspace; and

projecting the one or more exception events to the visualization model by mapping the position information of the one or more exception events to the pair of points in the visualization model.

18. The method of claim 11 , wherein defining the virtual model of the workspace further comprises:

scanning the workspace in the real-world three-dimensional environment via a camera system; and

generating a three-dimensional virtual model of the workspace based on data from the scan, the three-dimensional virtual model including the one or more virtual components that represent corresponding real-world objects in the workspace, the one or more virtual components having positions relative to the pair of anchor points.

19. The method of claim 18 , further comprising:

receiving a user alignment input to align a virtual component with a corresponding real object in the workspace; and

adjusting a position of the virtual component relative to the pair of anchor points based on the user alignment input.

20. A computer system comprising:

one or more processors configured to:

in a runtime phase:

identify a workflow including one or more telemetry events and a route in a workspace;

receive telemetry data from a telemetry device for a user performing the workflow;

compare the received telemetry data to the identified workflow;

determine one or more exception events indicating a deviation from the identified workflow, the one or more exception events including position information indicating a location of the exception event relative to the pair of anchor points in the workspace;

in a visualization phase:

determine a visualization model based on a virtual model of the workspace;

map a pair of points in the visualization model to the pair of anchor points in the virtual model of the workspace;

project the one or more exception events to the visualization model by mapping the position information of the one or more exception events to the pair of points in the visualization model; and

display the visualization model via a display of the computer system.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Apr 24, 2020
From: SMITH, JESSE MACDONALD; YOUNGSTROM, ERIC CHARLES; JACKSON, JEREMY BRENT; PAVLOVSKY, NICHOLAS JAMES; FIERST, JASON JOHN; ROSAS, DANIEL TAYLOR; JOHNSTON, GEORGE OLIVER; CHEN, TED TAI-YU
To: MICROSOFT TECHNOLOGY LICENSING, LLC
Reel/Frame 052489/0242 →
Cited By (5)
US 12,399,727 US 12,423,881 US 12,469,228 US 12,579,742 US 12,682,561