IP Library › Granted Patent US 9,710,700
Granted Patent B2
US 9,710,700 · App. 15/266,966 · Granted Jul 18, 2017

Systems and method for activity monitoring

Inventors: Maxime Jean Jerome Bilet (Seattle, WA); Roderick A. Hyde (Redmond, WA); Muriel Y. Ishikawa (Livermore, CA); Jordin T. Kare (San Jose, CA); Nathan P. Myhrvold (Bellevue, WA); Tony S. Pan (Bellevue, WA); Nels R. Peterson (Bellevue, WA); Clarence T. Tegreene (Mercer Island, WA); Lowell L. Wood, Jr. (Bellevue, WA); Victoria Y. H. Wood (Livermore, CA); Christopher Charles Young (Seattle, WA)
Assignee: ELWHA LLC
G06K9/00335G06F3/017G06F3/033G06F19/00G06F19/3431G06F19/3475G06K9/00624G06K9/00771G06Q10/0635G06Q30/018G06Q50/12G06K2209/17
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 9,710,700
App. No.
15/266,966
Granted
Jul 18, 2017
Kind
B2
Abstract

A system for activity monitoring includes a tracking component, an activity identification component, a procedural component, and a notification component. The tracking component tracks an individual and one or more objects in a work area using a three-dimensional tracking system. The activity identification component identifies an activity of the individual and any of the one or more objects that are affected by the activity. The procedural component determines whether the activity violates one or more procedural rules pertaining to one or more of the individual, the work area, and the one or more objects. The notification component provides a notification of a violation.

Claims (37)

1. A non-transitory computer-readable storage medium comprising program code for causing one or more processors to perform a method for monitoring an activity, the method comprising:

tracking an individual in a work area using a three-dimensional tracking system comprising at least a camera to sense a movement of a second portion of the individual that is in view of the camera and a processor to predict a position of a first portion of the individual that is out of view of the camera based on the movement of the second portion of the individual;

tracking one or more object in the work area using the three-dimensional tracking system and a sensor to obtain data corresponding to the one or more objects within the work area;

identifying, using an activity monitoring device comprising a computing device with a processor, an activity of the individual and any of the one or more objects that are affected by the activity based on at least one of the movement of the second portion of the individual, the predicted position of the first portion of the individual, and the data corresponding to the one or more objects obtained by the sensor;

determining, using the activity monitoring device comprising the computing device with the processor, whether the activity violates one or more procedural rules pertaining to one or more of the individual, the work area, and the one or more objects; and

providing a notification of a violation to an output device.

2. The computer-readable storage medium of claim 1 , wherein tracking the one or more objects comprises tracking a food item.

3. The computer-readable storage medium of claim 1 , wherein tracking the one or more objects comprises tracking tableware.

4. The computer-readable storage medium of claim 1 , wherein tracking the one or more objects comprises tracking a cleaning implement.

5. The computer-readable storage medium of claim 1 , wherein tracking an individual and one or more objects in a work area comprises tracking the individual and the one or more objects in a kitchen.

6. The computer-readable storage medium of claim 1 , wherein tracking comprises tracking an individual and one or more objects in a work area logically divided into two or more logical areas.

7. The computer-readable storage medium of claim 6 , wherein the two or more logical areas comprise a raw meat area.

8. The computer-readable storage medium of claim 6 , wherein the two or more logical areas comprise a hand washing area.

9. The computer-readable storage medium of claim 1 , wherein the identified activity comprises contact between the individual and an object of the one or more objects.

10. The computer-readable storage medium of claim 1 , wherein the identified activity comprises contact between an individual and a surface of the work area.

11. The computer-readable storage medium of claim 1 , wherein the identified activity comprises contact between an object of the one or more objects and a surface of the work area.

12. The computer-readable storage medium of claim 1 , wherein the identified activity comprises an activity having a risk of contaminating an object of the one or more objects.

13. The computer-readable storage medium of claim 1 , wherein the identified activity comprises an activity having a risk of contaminating at least a portion of the work area.

14. The computer-readable storage medium of claim 1 , wherein a procedural rule of the one or more procedural rules defines violation of a food handling procedure and determining comprises determining whether the activity violates the food handling procedure.

15. The computer-readable storage medium of claim 1 , wherein the procedural rule defines a combination of one or more work area conditions that comprise a violation and wherein determining whether the activity violates the procedural rule comprises comparing current work area conditions to the defined combination of work area conditions.

16. The computer-readable storage medium of claim 1 , wherein providing a notification comprises indicating in the work area an occurrence of a violation in response to determining that the activity violates a procedural rule of the one or more procedural rules.

17. The computer-readable storage medium of claim 16 , wherein indicating comprises visually indicating the occurrence of the violation.

18. The computer-readable storage medium of claim 17 , wherein visually indicating comprises spotlighting one or more of an object of the one or more objects, the individual, and a portion of the work area using a spotlight.

19. The computer-readable storage medium of claim 17 , wherein visually indicating comprises displaying on a display a diagram of the work area indicating the occurrence of the violation.

20. The computer-readable storage medium of claim 16 , wherein indicating comprises audibly playing a sound indicating the occurrence of the violation.

21. A system for activity monitoring, comprising:

a tracking component to track an individual in a work area using a three-dimensional tracking system comprising at least a camera to sense a movement of a second portion of the individual that is in view of the camera and a processor to predict a position of a first portion of the individual that is out of view of the camera based on the movement of the second portion of the individual, and track one or more objects in the work area using the three-dimensional tracking system;

a sensor to obtain data corresponding to the one or more objects in the work area;

a computing device with a processor, the computing device including:

an activity identification component to identify, an activity of the individual and any of the one or more objects that are affected by the activity based on at least one of the movement of the second portion of the individual, the predicted position of the first portion of the individual, and the data corresponding to the objects obtained by the sensor; and

a procedural component to determine whether the activity violates one or more procedural rules pertaining to one or more of the individual, the work area, and the one or more objects; and

a notification component to provide a notification of a violation using a visual output within the work area.

22. The system of claim 21 , further comprising a tracking storage component to store information regarding tracking of the individual and the one or more objects in the work area.

23. The system of claim 22 , wherein the tracking storage component stores video footage of the work area.

24. The system of claim 21 , wherein the three-dimensional tracking system comprises a tracer detection component to detect a tracer material, the tracer material comprising a material that transfers to a contacting surface in response to contact with the tracer material.

25. The system of claim 24 , wherein the three-dimensional tracking system comprises a tracer trigger component to trigger detectability of the tracer material.

26. The system of claim 21 , wherein the notification component provides the notification by automatically issuing a citation in response to the procedural component determining that the activity violates the one or more procedural rules.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Sep 15, 2016
From: BILET, MAXIME JEAN JEROME; HYDE, RODERICK A.; ISHIKAWA, MURIEL Y.; KARE, JORDIN T.; MYHRVOLD, NATHAN P.; PAN, TONY S.; PETERSON, NELS R.; TEGREENE, CLARENCE T.; WOOD, LOWELL L., JR.; WOOD, VICTORIA Y.H.; YOUNG, CHRISTOPHER CHARLES
To: ELWHA LLC
Reel/Frame 039760/0762 →
Continuity (2)
Continuation 13777548 · Feb 26, 2013
Related Publication 20170004357A1 · Jan 5, 2017