IP Library Granted Patent US 10,147,238
Granted Patent B2
US 10,147,238 · App. 15/381,588 · Granted Dec 4, 2018

Holographic technology implemented retail solutions

Inventors: Robert B. Locke (Sonoma, CA); Paul B. Rasband (Fremont, CA)
Assignee: Tyco Fire & Security GmbH
G06T19/006G06F3/16G06F17/218G06F17/241G06F17/2705G06Q30/0201G10L15/22G10L15/26G06F3/017G06T19/20G06T2219/004G10L15/30G10L2015/223
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,147,238
App. No.
15/381,588
Granted
Dec 4, 2018
Kind
B2
Abstract

Disclosed are techniques that use mixed reality, e.g., augmented reality and virtual reality technologies to improve analysis of retail processes and activity in retail establishments. A server system receives images from a mixed reality device, which images including representations of physical objects and receives voice data from the mixed reality device and executes a cognitive agent to process the voice data received from the mixed reality device that generates a set of virtual objects that include data describing merchandising process concepts regarding the set of physical objects, based on input queries received in the voice data and sending by a server system, the set of virtual objects to the mixed reality device.

Claims (48)

1. A system comprises:

a server system comprising:

one or more processor devices;

memory in communication with the one or more processor devices; and

a storage device that stores a program of computing instructions for execution by the processor using the memory, the program comprising instructions configured to cause the processor to:

receive images from a mixed reality device, which images including representations of physical objects;

receive voice data from the mixed reality device;

execute a cognitive agent to process the voice data received from the mixed reality device with the cognitive agent including:

a natural language process pre-processor that parses text input into tagged words and phrases to provide a first data structure that represents the text input with words and phrases connected to each other and annotated with indicators of meaning;

a response macro-planner that determines information content of a response from the representation of the text input to produce a second data structure that contains the response in a form that contains concepts and concept qualifiers that comprise logic behind the response; and

a micro-planner that produces sentences from the second data structure for the response from the cognitive agent;

determine location information to determine a correspondence between the representations of the physical objects in the images and stored information concerning the set of physical objects;

generate a set of virtual objects that include data describing merchandising process concepts regarding the set of physical objects, based on input queries received in the voice data; and

send the set of virtual objects to the mixed reality device.

2. The system of claim 1 comprises:

a voice to text translator.

3. The system of claim 1 wherein the processor configured to execute the cognitive agent is further configured to:

receive user inputs regarding the physical objects.

4. The system of claim 3 wherein the received user inputs regarding the physical objects are highlights applied to a selected one or more of the physical objects.

5. The system of claim 1 wherein the processor, configured to execute the cognitive agent, is further configured to:

analyze user inputs; and

resolve ambiguities in the user inputs by accessing data directly from the mixed reality device, which data specifies the physical objects by having a highlight applied on the specified one of the physical objects.

6. The system of claim 1 wherein the cognitive agent is configured to:

receive voice input in the form of a question regarding a product displayed in the mixed reality device;

generate a structure that represents an answer to the question in the voice input, by executing the cognitive agent to determine meaning of the question and forming queries to execute merchandizing algorithms that answer the question.

7. The system of claim 6 wherein the server system is further configured to:

send the answer to the mixed reality device.

8. The system of claim 6 wherein the server system is further configured to:

execute one or more merchandizing algorithms that apply the data derived from the question and that apply data corresponding to one or more merchandizing parameters associated with the set of physical objects to provide as the output data one or more merchandizing values associated with one or more of the physical objects in a view of the set of physical objects.

9. A computer implemented method, the method comprising:

receiving by a server system, images from a mixed reality device, which images including representations of physical objects;

receiving by a server system, voice data from the mixed reality device;

processing by a server system, the voice data received from the mixed reality device with a cognitive agent with processing further comprising:

parsing by a natural language process pre-processor text input into “tagged” words and phrases to provide a first data structure that represents text input with words and phrases connected to each other and annotated with indicators of meaning;

determining by a response macro-planner information content of a response from the representation of the text input to produce a second data structure that contains the response in a form that contains concepts and concept qualifiers that comprise logic behind the response; and

producing by a micro-planner sentences from the second data structure for the response from the cognitive agent;

determining by a server system, location information to determine a correspondence between the representations of the physical objects in the images and stored information concerning the set of physical objects;

generating by a server system, a set of virtual objects that include data describing merchandising process concepts regarding the set of physical objects, based on input queries received in the voice data; and

sending by a server system, the set of virtual objects to the mixed reality device.

10. The method of claim 9 , further comprising:

executing by the server system the cognitive agent.

11. The method of claim 9 , further comprising:

analyzing by the server system user inputs; and

resolving by the server system ambiguities in the user inputs by accessing data directly from the mixed reality device, which data specifies the physical objects by having a highlight applied on the specified one of the physical objects.

12. The method of claim 9 , further comprising:

receiving by the server system voice input in the form of a question regarding a product displayed in the mixed reality device;

generating by the server system a structure that represents an answer to the question in the voice data, by executing the cognitive agent to determine meaning of the question and forming queries to execute merchandizing algorithms that answer the question; and

executing by the server system one or more merchandizing algorithms that apply the data derived from the question and that apply data corresponding to one or more merchandizing parameters associated with the set of physical objects to provide as the output data one or more merchandizing values associated with one or more of the physical objects in a view of the set of physical objects.

Assignments (3)
NUNC PRO TUNC ASSIGNMENT Recorded Jan 29, 2024
From: TYCO FIRE & SECURITY GMBH
To: SENSORMATIC ELECTRONICS, LLC
Reel/Frame 066271/0678 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded May 11, 2018
From: LOCKE, ROBERT B.
To: TYCO FIRE & SECURITY GMBH
Reel/Frame 045777/0552 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Apr 20, 2018
From: RASBAND, PAUL B.
To: TYCO FIRE & SECURITY GMBH
Reel/Frame 045595/0274 →
Continuity (3)
Provisional Application 62361669 · Jul 13, 2016
Provisional Application 62361053 · Jul 12, 2016
Related Publication 20180018824A1 · Jan 18, 2018
Cited By (1)
US 12,488,360