Multi-person pose estimation using skeleton prediction
Embodiments provide functionality for identifying joints and limbs in images. An embodiment extracts features from an image to generate feature maps and, in turn, processes the feature maps using a single convolutional neural network trained based on a target model that includes joints and limbs. The processing generates both a directionless joint confidence map indicating confidence with which pixels in the image depict one or more joints and a directionless limb confidence map indicating confidence with which the pixels in the image depict one or more limbs between adjacent joints of the one or more joints, wherein adjacency of joints is provided by the target model. To continue, indications of the one or more joints and the one or more limbs in the image are generated using the directionless joint confidence map, the directionless limb confidence map, and the target model. Embodiments can be deployed on mobile and embedded systems.
1. A computer-implemented method of identifying joints and limbs in an image, the method comprising:
extracting features from an image to generate feature maps;
processing the feature maps using a single convolutional neural network formed of separable convolutional network elements and trained based on a target model that includes joints and a plurality of target limbs, the processing directly generating both a directionless joint confidence map indicating confidence with which pixels in the image depict one or more joints and a single directionless limb confidence map indicating confidence with which the pixels in the image depict one or more limbs, of the plurality of target limbs, between adjacent joints of the one or more joints, adjacency of joints being provided by the target model; and
generating indications of the one or more joints and the one or more limbs in the image using the directionless joint confidence map, the directionless limb confidence map, and the target model.
2. The computer-implemented method of claim 1 wherein the convolutional neural network is a first convolutional neural network and wherein the method further comprises:
concatenating the feature maps, the directionless joint confidence map, and the directionless limb confidence map;
processing the concatenated feature maps, directionless joint confidence map, and directionless limb confidence map using a second convolutional neural network trained based on the target model, the processing using the second convolutional neural network to generate both a refined directionless joint confidence map indicating confidence with which pixels in the image depict the one or more joints and a refined directionless limb confidence map indicating confidence with which the pixels in the image depict the one or more limbs; and
generating the indications of the one or more joints and the one or more limbs in the image using the refined directionless joint confidence map, the refined directionless limb confidence map, and the target model.
3. The computer-implemented method of claim 1 wherein the convolutional neural network has a single set of weights and the method further comprises:
training the convolutional neural network, based on the target model, by adjusting the single set of weights.
4. The computer-implemented method of claim 1 wherein the target model represents at least one of: a human, animal, machine, and robot.
5. The computer-implemented method of claim 1 wherein the target model indicates joints and limbs of a target object.
6. The computer-implemented method of claim 1 wherein the image contains multiple target objects, and further comprising:
generating indications of limbs and joints in the image for each of the multiple target objects.
7. The computer-implemented method of claim 6 further comprising:
generating respective indications of a pose of each of the multiple target objects.
8. The computer-implemented method of claim 1 wherein processing the feature maps using a single convolutional neural network further comprises:
generating a respective joint confidence map for each joint type in the image, each respective joint confidence map indicating confidence with which pixels in the image depict a respective joint type.
9. A computer system for identifying joints and limbs in an image, the computer system comprising:
a processor; and
a memory with computer code instructions stored thereon, the processor and the memory, with the computer code instructions, being configured to cause the system to:
extract features from an image to generate feature maps;
process the feature maps using a single convolutional neural network formed of separable convolutional network elements and trained based on a target model that includes joints and a plurality of target limbs, the processing directly generating both a directionless joint confidence map indicating confidence with which pixels in the image depict one or more joints and a single directionless limb confidence map indicating confidence with which the pixels in the image depict one or more limbs, of the plurality of target limbs, between adjacent joints of the one or more joints, adjacency of joints being provided by the target model; and
generate indications of the one or more joints and the one or more limbs in the image using the directionless joint confidence map, the directionless limb confidence map, and the target model.
10. The system of claim 9 wherein the convolutional neural network is a first convolutional neural network and wherein the processor and the memory, with the computer code instructions, are further configured to cause the system to:
concatenate the feature maps, the directionless joint confidence map, and the directionless limb confidence map;
process the concatenated feature maps, directionless joint confidence map, and directionless limb confidence map using a second convolutional neural network trained based on the target model, the processing using the second convolutional neural network to generate both a refined directionless joint confidence map indicating confidence with which pixels in the image depict the one or more joints and a refined directionless limb confidence map indicating confidence with which the pixels in the image depict the one or more limbs; and
generate the indications of the one or more joints and the one or more limbs in the image using the refined directionless joint confidence map, the refined directionless limb confidence map, and the target model.
11. The system of claim 9 wherein the convolutional neural network has a single set of weights and the processor and the memory, with the computer code instructions, are further configured to cause the system to:
train the convolutional neural network, based on the target model, by adjusting the single set of weights.
12. The system of claim 9 wherein the target model represents at least one of: a human, animal, machine, and robot.
13. The system of claim 9 wherein the target model indicates joints and limbs of a target object.
14. The system of claim 9 wherein the image contains multiple target objects and wherein the processor and the memory, with the computer code instructions, are further configured to cause the system to:
generate indications of limbs and joints in the image for each of the multiple target objects.
15. The system of claim 14 wherein the processor and the memory, with the computer code instructions, are further configured to cause the system to:
generate respective indications of a pose of each of the multiple target objects.
16. The system of claim 9 wherein, in processing the feature maps using a single convolutional neural network, the processor and the memory, with the computer code instructions, are further configured to cause the system to:
generate a respective joint confidence map for each joint type in the image, each respective joint confidence map indicating confidence with which pixels in the image depict a respective joint type.
17. A computer program product for identifying joints and limbs in an image, the computer program product comprising: one or more non-transitory computer-readable storage devices and program instructions stored on at least one of the one or more storage devices, the program instructions, when loaded and executed by a processor, cause an apparatus associated with the processor to: extract features from an image to generate feature maps; process the feature maps using a single convolutional neural network formed of separable convolutional network elements and trained based on a target model that includes joints and a plurality of target limbs, the processing directly generating both a directionless joint confidence map indicating confidence with which pixels in the image depict one or more joints and a single directionless limb confidence map indicating confidence with which the pixels in the image depict one or more limbs, of the plurality of target limbs, between adjacent joints of the one or more joints, adjacency of joints being provided by the target model; and generate indications of the one or more joints and the one or more limbs in the image using the directionless joint confidence map, the directionless limb confidence map, and the target model.
18. The computer program product of claim 17 where the program instructions, when loaded and executed by the processor, further cause the apparatus associated with the processor to:
concatenate the feature maps, the directionless joint confidence map, and the directionless limb confidence map;
process the concatenated feature maps, directionless joint confidence map, and directionless limb confidence map using a second convolutional neural network trained based on the target model, the processing using the second convolutional neural network to generate both a refined directionless joint confidence map indicating confidence with which pixels in the image depict the one or more joints and a refined directionless limb confidence map indicating confidence with which the pixels in the image depict the one or more limbs; and
generate the indications of the one or more joints and the one or more limbs in the image using the refined directionless joint confidence map, the refined directionless limb confidence map, and the target model.