Recognizing and tracking poses using digital imagery captured from multiple fields of view
Poses or gestures of actors within a scene may be detected and tracked using multiple imaging devices aligned with fields of view that overlap at least in part. Images captured by the imaging devices may be synchronized and provided to a classifier to recognize body parts within the images, and score maps indicative of locations of peak probabilities that the images include the respective body parts may be generated. Locations of peak values within the score maps may be correlated with one another to confirm that a given body part is depicted in two or more fields of view, and vectors indicative of distances to or ranges of motion of body parts, with respect to the given body part, may be generated. Motion of the body parts may be tracked in subsequent images, and a virtual model of the body parts may be generated and updated based on the motion.
1. A system for tracking multi-joint subjects in an area of real space, comprising:
a plurality of cameras, cameras in the plurality of cameras producing respective sequences of images of corresponding fields of view in the real space, the field of view of each camera overlapping with the field of view of at least one other camera in the plurality of cameras;
a processing system coupled to the plurality of cameras, the processing system comprising:
at least a first component configured to receive the sequences of images from the plurality of cameras, wherein the first component is further configured to process images to generate corresponding arrays of joint data structures, the arrays of joint data structures corresponding to particular images classifying elements of the particular images by joint type, time of the particular image, and coordinates of the element in the particular image;
at least a second component configured to receive the arrays of joint data structures corresponding to images in sequences of images from cameras having overlapping fields of view, wherein the second component is further configured to translate the coordinates of the elements in the arrays of joint data structures corresponding to images in different sequences into candidate joints having coordinates in real space; and
at least a third component configured to identify sets of candidate joints having coordinates in real space as multi-joint subjects in the real space.
2. The system of claim 1 , wherein at least the first component comprises a convolutional neural network.
3. The system of claim 1 , wherein at least the first component processes images to generate arrays for elements of the image, and wherein an array for a particular element of an image includes values for a plurality of joint types for the particular element, and to select a joint type for the joint data structure of the particular element based on the array.
4. The system of claim 1 , wherein the third component is configured to execute one or more functions based on physical relationships among joints of subjects in real space to identify sets of candidate joints as multi-joint subjects.
5. The system of claim 4 , wherein the processing system further comprises:
at least a fourth component configured to store the sets of joints identified as multi-joint subjects,
wherein the third component is further configured to determine whether a candidate joint identified in images taken at a particular time corresponds with a member of one of the sets of candidate joints identified as multi-joint subjects in preceding images.
6. The system of claim 1 , wherein cameras in the plurality of cameras are configured to generate synchronized sequences of images.
7. The system of claim 1 , wherein the plurality of cameras comprise cameras disposed over and having fields of view encompassing respective parts of the area in real space, and the coordinates in real space of members of a set of candidate joints identified as a multi-joint subject identify locations in the area of the multi-joint subject.
8. The system of claim 1 , wherein the processing system further comprises:
at least a fourth component configured to track locations of a plurality of multi-joint subjects in the area of real space.
9. The system of claim 8 , wherein the processing system further comprises:
at least a fifth component configured to determine when multi-joint subjects in the plurality of multi-joint subjects leave the area of real space.
10. The system of claim 1 , wherein the processing system further comprises:
at least a fourth component configured to track locations in the area of real space of multiple candidate joints that are members of a set of candidate joints identified as a particular multi-joint subject.
11. A method for tracking multi-joint subjects in an area of real space, comprising:
using a plurality of cameras to produce respective sequences of images of corresponding fields of view in the real space, the field of view of each camera overlapping with the field of view of at least one other camera in the plurality of cameras;
processing images in the sequences of images to generate corresponding arrays of joint data structures, the arrays of joint data structures corresponding to particular images classifying elements of the particular images by joint type, time of the particular image, and coordinates of the element in the particular image;
translating the coordinates of the elements in the arrays of joint data structures corresponding to images in different sequences into candidate joints having coordinates in the real space; and
identifying sets of candidate joints having coordinates in real space as multi-joint subjects in the real space.
12. The method of claim 11 , wherein said processing images includes using convolutional neural networks.
13. The method of claim 11 , wherein said processing images includes generating confidence arrays for elements of the image, and wherein a confidence array for a particular element of an image includes confidence values for a plurality of joint types for the particular element, and selecting a joint type for the joint data structure of the particular element based on the confidence array.
14. The method of claim 11 , wherein identifying sets of candidate joints comprises applying functions based on physical relationships among joints of subjects in real space to identify sets of candidate joints as multi-joint subjects.
15. The method of claim 14 , including storing the sets of joints identified as multi-joint subjects, and wherein the identifying sets of candidate joints includes determining whether a candidate joint identified in images taken at a particular time corresponds with a member of one of the sets of candidate joints identified as a multi-joint subject in a preceding image.
16. The method of claim 11 , wherein the sequences of images are synchronized.
17. The method of claim 11 , wherein the plurality of cameras comprise cameras disposed over and having fields of view encompassing respective parts of the area in real space, and the coordinates in real space of members of a set of candidate joints identified as a multi-joint subject identify locations in the area of the multi-joint subject.
18. The method of claim 11 , including tracking locations of a plurality of multi-joint subjects in the area of real space.
19. The method of claim 18 , including determining when a multi-joint subject in the plurality of multi-joint subjects leaves the area of real space.
20. The method of claim 11 , including tracking locations in the area of real space of multiple candidate joints that are members of a set of candidate joints identified as a particular multi-joint subject.