IP Library › Granted Patent US 8,811,719
Granted Patent B2
US 8,811,719 · App. 13/098,387 · Granted Aug 19, 2014

Inferring spatial object descriptions from spatial gestures

Inventors: Andrew David Wilson (Redmond, WA); Christian Holz (Berlin, DE)
Assignee: Microsoft Corporation
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 8,811,719
App. No.
13/098,387
Granted
Aug 19, 2014
Kind
B2
Abstract

Three-dimensional (3-D) spatial image data may be received that is associated with at least one arm motion of an actor based on free-form movements of at least one hand of the actor, based on natural gesture motions of the at least one hand. A plurality of sequential 3-D spatial representations that each include 3-D spatial map data corresponding to a 3-D posture and position of the hand at sequential instances of time during the free-form movements may be determined, based on the received 3-D spatial image data. An integrated 3-D model may be generated, via a spatial object processor, based on incrementally integrating the 3-D spatial map data included in the determined sequential 3-D spatial representations and comparing a threshold time value with model time values indicating numbers of instances of time spent by the hand occupying a plurality of 3-D spatial regions during the free-form movements.

Claims (70)

1. A system comprising:

at least one device processor, and including a machine readable storage device storing executable instructions, the executable instructions configured, when executed, to cause the at least one device processor to:

initiate access to a database including a plurality of database objects, each database object associated with a respective one of a plurality of predefined three-dimensional (3-D) models, each of the 3-D models simulating an appearance of a corresponding one of a plurality of predetermined 3-D items;

receive 3-D spatial image data associated with at least one arm motion of an actor based on free-form movements of at least one hand of the actor, based on natural gesture motions;

generate an integrated 3-D model based on incrementally aggregating temporally sequential 3-D representations of the 3-D positioning of the at least one hand in accordance with the received 3-D spatial image data, selecting portions of the received 3-D spatial image data for aggregation based on comparing a threshold time value with numbers of instances of time spent by the at least one hand within a plurality of 3-D regions during the free-form movements; and

select at least one of the predetermined 3-D items, based on accessing the database and determining at least one of the database objects that is associated with at least one of the predefined 3-D models that matches the integrated 3-D model.

2. The system of claim 1 , wherein the executable instructions are configured, when executed, to cause the at least one device processor to:

obtain an update 3-D model that simulates an appearance of a predefined update 3-D item and initiate storage, in the database, of an update database object associated with the update 3-D model.

3. The system of claim 2 , wherein the predefined update 3-D item includes one or more of:

a 3-D inventory item, a 3-D gaming object, a 3-D real-world item, or a 3-D virtual reality environment object.

4. The system of claim 2 , wherein the executable instructions are configured, when executed, to cause the at least one device processor to obtain the update 3-D model based on one or more of:

receiving image data associated with a picture of the predefined update 3-D item, or

receiving the update 3-D model via an input device.

5. The system of claim 1 , wherein the executable instructions are configured, when executed, to cause the at least one device processor to:

receive audio data associated with at least one verbal indicator representing an utterance, wherein

select, at least one of the predetermined 3-D items, based on accessing the database and determining at least one of the database objects that is associated with at least one of the predefined 3-D models that matches the integrated 3-D model, based on the verbal indicator.

6. The system of claim 1 , wherein the executable instructions are configured, when executed, to cause the at least one device processor to:

initialize a virtual 3-D map space based on discretized 3-D virtual map elements represented as volumized elements that each include weight values initialized to an initial value, wherein

the virtual 3-D map space represents a 3-D space in close proximity to the actor, and wherein

the executable instructions are configured, when executed, to cause the at least one device processor to:

proportionally increment the weight values of selected volumized elements associated with 3-D regions of the 3-D space, based on a determination indicating that a portion of the at least one hand has occupied the 3-D regions over a period of time during the free-form movements, and

compare the threshold value with the weight values of each of the volumized elements.

7. The system of claim 6 , wherein the executable instructions are configured, when executed, to cause the at least one device processor to:

determine a depth, position, and orientation of the at least one hand,

determine locations of volumized elements associated with the virtual 3-D map space that correspond to the depth, position, and orientation of the at least one hand, and

activate a plurality of the volumized elements associated with an area of the virtual 3-D map space that represents a depth, location, and orientation corresponding to the depth, location, and orientation of the at least one hand, based on the determined locations.

8. The system of claim 7 , wherein:

the volumized elements include volumetric picture elements (voxels);

the executable instructions are configured, when executed, to cause the at least one device processor to initialize the voxels to an inactive state, with initialized weight values; and

the executable instructions are configured, when executed, to cause the at least one device processor to activate groups of voxels based on the determination indicating that a portion of the at least one hand has occupied the 3-D regions over a period of time during the free-form movements, based on increasing the weight values associated with the groups of voxels with each activation of the groups of voxels.

9. The system of claim 6 , wherein the executable instructions are configured, when executed, to cause the at least one device processor to:

determine an enclosed space indicated by a posture of the at least one hand,

determine a depth of the enclosed space based on a determination of a depth of an area surrounding the enclosed space, and

activate a plurality of the volumized elements associated with an area of the virtual 3-D space that represents a depth, location, and orientation corresponding to the enclosed space, in lieu of activation of a plurality of the volumized elements associated with an area of the virtual 3-D space that represents a depth, location, and orientation corresponding to the depth, location, and orientation of the at least one hand, during a period of time associated with the posture indicating the enclosed space.

10. The system of claim 1 , wherein the executable instructions are configured, when executed, to cause the at least one device processor to:

generate a first alignment of one of the predefined 3-D models and the integrated 3-D model based on matching at least one component that is included in the one of the predefined 3-D models and the integrated 3-D model, based on scaling, translating, and rotating the one of the predefined 3-D models and the integrated 3-D model; and

generate a second alignment of the one of the predefined 3-D models and the integrated 3-D model based on an iterative closest point algorithm, based on the first alignment.

11. The system of claim 1 , wherein the executable instructions are configured, when executed, to cause the at least one device processor to:

generate a first alignment of one of the predefined 3-D models and the integrated 3-D model based on matching at least one component that is included in the one of the predefined 3-D models and the integrated 3-D model, based on scaling, translating, and rotating the one of the predefined 3-D models and the integrated 3-D model; and

generate a second alignment of the one of the predefined 3-D models and the integrated 3-D model based on a brute force alignment that includes a plurality of scales, rotations, and translations of the one of the predefined 3-D models and the integrated 3-D model, based on the first alignment.

12. A computer program product tangibly embodied on a machine readable storage device and including executable code that, when executed, is configured to cause at least one data processing apparatus to:

receive three-dimensional (3-D) sensor data associated with at least one natural gesture of an actor based on free-form movements of the actor, based on natural gesture motions emulating an appearance of a predetermined three-dimensional (3-D) item;

generate an integrated 3-D model based on aggregating received 3-D sensor data that represents 3-D positioning of at least one 3-D moving object associated with the actor in accordance with the free-form movements, selecting portions of the received 3-D sensor data for aggregation based on comparing a threshold time value with numbers of instances of time spent by the at least one 3-D moving object within a plurality of 3-D regions during the free-form movements; and

determine, via a spatial object processor, a predefined 3-D model associated with a database object that matches the integrated 3-D model.

13. The computer program product of claim 12 , wherein the executable code, when executed, is configured to cause the at least one data processing apparatus to:

select portions of the received 3-D sensor data for aggregation based on comparing the threshold time value with values indicating lengths of time spent by the at least one 3-D moving object within a plurality of 3-D regions during the free-form movements.

14. The computer program product of claim 12 , wherein the executable code, when executed, is configured to cause the at least one data processing apparatus to:

obtain an update 3-D model that simulates an appearance of a predefined update 3-D item and initiate storage, in a database, of an update database object associated with the update 3-D model.

15. The computer program product of claim 14 , wherein the executable code, when executed, is configured to cause the at least one data processing apparatus to:

obtain the update 3-D model based on one or more of:

receiving image data associated with a picture of the predefined update 3-D item, or

receiving the update 3-D model via an input device.

16. The computer program product of claim 12 , wherein the executable code, when executed, is configured to cause the at least one data processing apparatus to:

receive audio data associated with at least one verbal indicator representing an utterance; and

select the predefined 3-D model based on accessing a database and determining the database object that matches the integrated 3-D model, based on the verbal indicator.

17. A computer program product tangibly embodied on a machine readable storage device and including executable code that, when executed, is configured to cause at least one data processing apparatus to:

initiate access to a database including a plurality of database objects, each database object associated with a respective one of a plurality of predefined three-dimensional (3-D) models, each of the 3-D models simulating an appearance of a corresponding one of a plurality of predetermined 3-D items;

receive 3-D spatial image data associated with at least one arm motion of an actor based on free-form movements of at least one hand of the actor, based on natural gesture motions;

generate an integrated 3-D model based on incrementally aggregating temporally sequential 3-D representations of the 3-D positioning of the at least one hand in accordance with the received 3-D spatial image data, selecting portions of the received 3-D spatial image data for aggregation based on comparing a threshold time value with numbers of instances of time spent by the at least one hand within a plurality of 3-D regions during the free-form movements; and

select, via a device processor, at least one of the predetermined 3-D items, based on accessing the database and determining at least one of the database objects that is associated with at least one of the predefined 3-D models that matches the integrated 3-D model.

18. The computer program product of claim 17 , wherein the executable code, when executed, is configured to cause the at least one data processing apparatus to:

obtain an update 3-D model that simulates an appearance of a predefined update 3-D item and initiate storage, in the database, of an update database object associated with the update 3-D model.

19. The computer program product of claim 17 , wherein the executable code, when executed, is configured to cause the at least one data processing apparatus to:

initialize a virtual 3-D map space based on discretized 3-D virtual map elements represented as volumized elements that each include weight values initialized to an initial value, wherein the virtual 3-D map space represents a 3-D space in close proximity to the actor;

proportionally increment the weight values of selected volumized elements associated with 3-D regions of the 3-D space, based on a determination indicating that a portion of the at least one hand has occupied the 3-D regions over a period of time during the free-form movements; and

compare the threshold value with the weight values of each of the volumized elements.

20. The computer program product of claim 17 , wherein the executable code, when executed, is configured to cause the at least one data processing apparatus to:

determine a depth, position, and orientation of the at least one hand,

determine locations of volumized elements associated with the virtual 3-D map space that correspond to the depth, position, and orientation of the at least one hand, and

activate a plurality of the volumized elements associated with an area of the virtual 3-D map space that represents a depth, location, and orientation corresponding to the depth, location, and orientation of the at least one hand, based on the determined locations of the volumized elements associated with the virtual 3-D map space that correspond to the depth, position, and orientation of the at least one hand.

Assignments (2)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Dec 9, 2014
From: MICROSOFT CORPORATION
To: MICROSOFT TECHNOLOGY LICENSING, LLC
Reel/Frame 034544/0001 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded May 2, 2011
From: WILSON, ANDREW DAVID; HOLZ, CHRISTIAN
To: MICROSOFT CORPORATION
Reel/Frame 026206/0092 →
Continuity (1)
Related Publication 20120275686A1 · Nov 1, 2012