Device for identifying and tracking multiple humans over time
A system recognizes human beings in their natural environment, without special sensing devices attached to the subjects, uniquely identifies them and tracks them in three dimensional space. The resulting representation is presented directly to applications as a multi-point skeletal model delivered in real-time. The device efficiently tracks humans and their natural movements by understanding the natural mechanics and capabilities of the human muscular-skeletal system. The device also uniquely recognizes individuals in order to allow multiple people to interact with the system via natural movements of their limbs and body as well as voice commands/responses.
1. A system for analyzing at least one user in a capture area, comprising:
a processor; and
a memory communicatively coupled to the processor when the system is operational, the memory bearing processor-executable instructions that, when executed on the processor, cause the system to at least:
receive depth data for the capture area;
receive color data for the capture area;
determine a location of the user in the capture area, including a location of a first body part and a second body part of the user, based on the depth data; and
determine an orientation of the first body part of the user relative to the second body part of the user at the location in the capture area based on the color data.
2. The system of claim 1 wherein the orientation of the user comprises the user facing toward a capture device that captures the color data, or the user facing away from the capture device.
3. The system of claim 1 wherein the instructions that, when executed on the processor, cause the system to at least determine the orientation of the user at the location in the capture area based on the color data, further cause the system to at least:
determine that the orientation of the user comprises the user facing toward a capture device in response to determining that the user's face is present in the color data.
4. The system of claim 1 , wherein the memory further bears instructions that, when executed on the processor, cause the system to at least: generate a skeletal model of the user based at least in part on the depth data.
5. The system of claim 4 , wherein the a memory further bears instructions that, when executed on the processor, cause the system to at least: track movement of the user over time based on the skeletal model.
6. The system of claim 5 , wherein when the memory further bears instructions that, when executed on the processor, cause the system to at least: determine that the system is unable to track movement of the user from the depth data without the color data; and
track movement of the user based on the depth data supplemented with the color data.
7. The system of claim 4 , wherein the memory further bears instructions that, when executed on the processor, cause the system to at least: track the user over time based on known mechanics of the human muscular-skeletal system.
8. The system of claim 1 wherein the instructions that, when executed on the processor, cause the system to at least determine the orientation of the user at the location in the capture area based on the color data, further cause the system to at least:
determine that the orientation of the user comprises the user facing away from a capture device in response to determining that a threshold amount of the user's hair color or less than a threshold amount of the user's facial features is present in the color data.
9. The system of claim 1 , wherein the color data comprises black and white or grayscale colors.
10. A method for tracking a human user in a capture area, comprising:
receiving depth data for the capture area;
receiving color data for the capture area;
determining a location of the user in the capture area, including a location of a first body part and a second body part of the user, based on the depth data; and
determining an orientation of the first body part of the user relative to the second body part of the user at the location in the capture area based on the color data.
11. The method of claim 10 , wherein the orientation of the user comprises the user facing toward a capture device that captures the color data, or the user facing away from the capture device.
12. The method of claim 10 , further comprising receiving sound data from at least one microphone, and recognizing and tracking comprises recognizing and tracking movement of the user in the capture area over time based on a combination of at least two of the depth data, the color data and the sound data.
13. The method of claim 10 , further comprising generating a skeletal model of the user from the depth data, and using the skeletal model to track movement of the user over time based on the known mechanics of the human muscular-skeletal system.
14. The method of claim 10 , further comprising:
determining that the system is unable to track movement of the user from the depth data without the color data; and
tracking movement of the user based on the depth data supplemented with the color data.
15. The method of claim 14 , further comprising generating a motion capture file of the movements of the user in real-time based on the tracked-movements of the user.
16. The method of claim 10 , further comprising using sound information provided by a microphone to distinguish between two users detected in the capture area based on recognition of each user's voice.
17. A computer-readable storage medium on which is stored computer-executable instructions for tracking a human user in a capture area, the instructions, when executed by a processor, causing the processor to perform operations comprising:
receive depth data for the capture area;
receive color data for the capture area;
determining a location of the user in the capture area, including a location of a first body part and a second body part of the user, based on the depth data; and
determining an orientation of the first body part of the user relative to the second body part of the user at the location in the capture area based on the color data.
18. The method of claim 10 , further comprising:
determining a constraint of where the first body part may be located based on a rotational orientation of the first body part relative to the second body part and the location of the second body part;
determining that the location of the first body part violates the constraint; and
in response to determining that the location of the first body part violates the constraint, adjusting the position of the first body part to a location that does not violate the constraint.
19. The method of claim 10 , further comprising:
determining a rotational orientation of the first body part relative to the second body part based on the color data, the rotational orientation indicating how the first body part is rotated at its location in the capture area.
20. The method of claim 10 , wherein the first body part and the second body part are both directly connected at a joint.