IP Library › Granted Patent US 10,417,878

Granted Patent B2

US 10,417,878 · App. 14/663,190 · Granted Sep 17, 2019

Method, computer program product, and system for providing a sensor-based environment

Inventors: Dean Frederick Herring (Youngsville, NC); Brad Matthew Johnson (Raleigh, NC); Ankit Singh (Morrisville, NC); Jeffrey John Smith (Raleigh, NC); Adrian Xavier Rodriquez (Raleigh, NC)

Assignee: Toshiba Global Commerce Solutions Holdings Corporation

G07G1/0054 G01B11/00 G01G19/4144 G01S19/13 G06F1/163 G06F3/012 G06F3/013 G06K9/00221 G06K9/00255 G06K9/00288 G06K9/00362 G06K9/00536 G06K9/00577 G06K9/00624 G06K9/209 G06K9/46 G06K9/6201 G06K9/6267 G06K9/78 G06Q10/087 G06Q20/18 G06Q20/204 G06Q20/208 G06Q20/209 G06Q20/322 G06Q30/016 G06Q30/0224 G06Q30/0235 G06Q30/0251 G06Q30/0269 G06Q30/0609 G06Q30/0623 G06Q30/0629 G06Q30/0631 G06Q30/0633 G06Q30/0635 G06Q30/0639 G06T7/66 G06T11/60 G07G1/0045 G07G1/0063 G07G1/0072 H04N5/225 H04N5/232 H04N7/181 G01B11/02 G01G19/40 G06K2009/00328 G06K2209/17 G06T2207/30232

View Patent ↗

Loading inventors, assignments & file history…

Monitor This Case

Get email alerts when status or documents change.

Order Certified Copies

Most orders are placed with the USPTO same day — all within 24 business hours.

Order via The Patent Place →

Pre-filled with this patent's details

Quick Facts

Patent No.: US 10,417,878
App. No.: 14/663,190
Granted: Sep 17, 2019
Kind: B2

External Links

USPTO Patent Center↗ Full Patent PDF↗

Abstract

Method, computer program product, and system to trigger one or more events during an interaction of a person with an item and within an environment having a plurality of items. The method includes acquiring, using a first visual sensor disposed within the environment, image information including the item and identifying the item using the image information. The method further includes acquiring, using a second visual sensor disposed within the environment, behavior information for the person relative to the identified item, analyzing the behavior information to identify at least a first behavior of the person, and classifying the first behavior into one of a number of predefined behavior types. The method further includes performing, based on the identified item and the classified first behavior, one or more predefined events.

Claims (75)

1. A computer-implemented method to trigger one or more events during an interaction of a person with an item and within an environment having a plurality of items, the method comprising:

acquiring, using a first visual sensor disposed within the environment, image information including the item;

identifying the item using the image information;

acquiring, using a second visual sensor disposed within the environment, behavior information for the person relative to the identified item, the behavior information comprising one or more images;

analyzing, using one or more computer processors, the behavior information to identify at least a first behavior of the person, the analysis comprising:

analyzing a first image of the one or more images, using the computer processors, to identify a body part of the person;

determining, using the computer processors, an angle formed by a position of the body part relative to a reference direction;

analyzing a second image of the one or more images, using the computer processors and based at least in part on the angle, to identify a portion of the second image comprising a field of view (FOV) of the person;

analyzing the identified portion of the second image to determine that the item falls within the FOV of the person; and

determining, based on the behavior information, a duration of time the identified item falls within the FOV of the person;

determining, using the one or more computer processors and based, at least in part, on the duration of time, that the first behavior corresponds to one of a number of predefined behavior types, and in response classifying the first behavior into the one of a number of predefined behavior types based, at least in part, on the duration of time;

performing, based on the identified item and the classified first behavior, one or more predefined events;

analyzing, using one or more computer processors, the behavior information to identify a second behavior of the person; and

determining, using the one or more computer processors, that the second behavior does not correspond to a predefined behavior type, and in response characterizing the second behavior, based on the behavior information, and storing the characterized second behavior.

2. The computer-implemented method of claim 1 , wherein the one or more predefined events includes one or more of updating a personal profile of the person, presenting information to the person, and updating environmental data.

3. The computer-implemented method of claim 1 , wherein the predefined behavior types are specified by two or more providers and prioritized based on a corresponding provider.

4. The computer-implemented method of claim 1 , further comprising:

validating the stored second behavior; and

associating the validated behavior with one or more predefined events.

5. The computer-implemented method of claim 1 , wherein the first behavior signals a beginning of an interaction period of the person with the item, and further comprising:

during the interaction period, identifying at least a third behavior of the person.

6. The computer-implemented method of claim 5 , wherein the third behavior signals an end of the interaction period, and further comprising:

responsive to identifying the third behavior, presenting at least one of recommendation information and promotion information to the person.

7. The computer-implemented method of claim 1 , wherein the body part comprises at least one of a head of the person, a shoulder of the person, or an eye of the person.

8. A computer program product to trigger one or more events during an interaction of a person with an item and within an environment having a plurality of items, the computer program product comprising:

a computer-readable storage medium having computer-readable program code embodied therewith, the computer-readable program code executable by one or more computer processors to perform an operation that includes:

acquiring, using a first visual sensor disposed within the environment, image information including the item;

identifying the item using the image information;

acquiring, using a second visual sensor disposed within the environment, behavior information for the person relative to the identified item, the behavior information comprising one or more images;

analyzing the behavior information to identify at least a first behavior of the person, the analysis comprising:

analyzing a first image of the one or more images to identify a body part of the person;

determining an angle formed by a position of the body part relative to a reference direction;

analyzing a second image of the one or more images, based at least in part on the angle, to identify a portion of the second image comprising a field of view (FOV) of the person;

analyzing the identified portion of the second image to determine that the item falls within the FOV of the person; and

determining, based on the behavior information, a duration of time the identified item falls within the FOV of the person;

determining, based, at least in part, on the duration of time, that the first behavior corresponds to one of a number of predefined behavior types, and in response classifying the first behavior into the one of a number of predefined behavior types based, at least in part, on the duration of time;

performing, based on the identified item and the classified first behavior, one or more predefined events;

analyzing the behavior information to identify a second behavior of the person; and

determining that the second behavior does not correspond to a predefined behavior type, and in response characterizing the second behavior, based on the behavior information, and storing the characterized second behavior.

9. The computer program product of claim 8 , wherein the one or more predefined events includes one or more of updating a personal profile of the person, presenting information to the person, and updating environmental data.

10. The computer program product of claim 8 , wherein the predefined behavior types are specified by two or more providers and prioritized based on a corresponding provider.

11. The computer program product of claim 8 , wherein the operation further includes:

validating the stored second behavior; and

associating the validated behavior with one or more predefined events.

12. The computer program product of claim 8 , wherein the first behavior signals a beginning of an interaction period of the person with the item, and wherein the operation further includes:

during the interaction period, identifying at least a third behavior of the person.

13. The computer program product of claim 12 , wherein the third behavior signals an end of the interaction period, and wherein the operation further includes:

responsive to identifying the third behavior, presenting at least one of recommendation information and promotion information to the person.

14. The computer program product of claim 8 , wherein the body part comprises at least one of a head of the person, a shoulder of the person, or an eye of the person.

15. A system to trigger one or more events during an interaction of a person with an item and within an environment having a plurality of items, the system comprising:

one or more computer processors;

at least first and second visual sensors disposed within the environment and communicatively coupled with the one or more computer processors; and

a memory containing program code which, when executed by the one or more computer processors, performs an operation that includes:

acquiring, using the first visual sensor, image information including the item;

identifying the item using the image information;

acquiring, using the second visual sensor, behavior information for the person relative to the identified item, the behavior information comprising one or more images;