Infant 2D pose estimation and posture detection system
Methods are provided for estimating a pose of an infant using image analysis and artificial intelligence. A classifier is trained using a dataset containing hybrid synthetic and real infant pose data. Multi-stage invariant representation machine learning strategies are employed that transfer knowledge from adjacent domains of adult poses and synthetic infant images into a fine-tuned domain-adapted infant pose estimation model.
1 . A system for estimating a pose of an infant, the system comprising:
one or more processors; and
at least one memory including instructions for a pose estimator, the pose estimator comprising a trained model for estimating infant poses trained on an adult pose dataset and an augmented dataset containing real infant pose data and synthetic infant pose data using domain adaptation to align features of the synthetic infant pose data with the real infant pose data; and
wherein the at least one memory further includes instructions that, when executed by the one or more processors, cause the system to:
receive one or more images of an infant transmitted from an imaging device; and
determine a pose of the infant using the trained model.
2 . The system of claim 1 further comprising one or more imaging devices for acquiring images of the infant.
3 . The system of claim 1 , wherein the augmented dataset is provided by a process comprising:
fitting infant model pose and shape parameters into a real infant image to form a reposed model, the infant model pose and shape parameters including body joints and shape coefficients; and
generating synthetic output images as the synthetic infant pose data.
4 . The system of claim 3 , wherein the synthetic output images are generated by an imaging process as a function of the infant model pose and shape parameters, imaging device parameters, body texture maps, and background images;
wherein the imaging device parameters include principal point and focal length of a camera; and
wherein the body texture maps include infant textures from infant clothing images augmented with adult textures from adult clothing images.
5 . The system of claim 3 , wherein the synthetic output images are augmented by minimizing a cost function, wherein the cost function is a sum of loss terms weighted by imaging device parameters, the loss terms including:
a joint-based data term comprising a distance between ground truth two dimensional joints and a two dimensional projection of corresponding posed three dimensional joints of the infant model pose for each joint;
a mixture of Gaussian pose priors learned from adult poses;
a shape penalty comprising a distance between a shape prior of the infant model and the shape parameters being optimized; and
a pose prior penalizing elbows and knees.
6 . The system of claim 1 , wherein the pose estimator further comprises:
a pose estimation component comprising a feature extractor and a pose predictor; and
a domain confusion component in communication with the pose estimation component to share the feature extractor and including a domain classifier operative to distinguish whether a feature on an input image belongs to a real image or synthetic image.
7 . The system of claim 6 , wherein the pose estimation component further comprises a residual neural network as an encoder and a pose head estimator as a decoder.
8 . The system of claim 6 , wherein in a first stage, the domain confusion component is fine-tuned using real infant pose data and synthetic infant pose data, to obtain a domain classifier for predicting whether features of an input image are from a synthetic infant image or a real infant image, using an optimization of a loss function to determine a binary cross entropy loss.
9 . The system of claim 8 , wherein in the first stage, the pose estimation component is locked.
10 . The system of claim 9 , wherein in a second stage, the pose estimation component is fine-tuned using the domain classifier to extract body keypoint information independently of differences between a real domain and a synthetic domain.
11 . The system of claim 10 , wherein in the second stage, weight of real data and synthetic data is balanced by maximizing a domain loss function, wherein features representing the real domain and features representing the synthetic domain become more similar.
12 . The system of claim 6 , wherein the feature extractor and the domain classifier are operative to enforce mapping of input images in a real domain or a synthetic domain into a same feature space after feature extraction.
13 . The system of claim 6 , wherein the domain classifier comprises a binary classifier with three fully connected layers operative to distinguish whether an input feature belongs to a real image or a synthetic image.
14 . The system of claim 6 , wherein the pose estimation component is pre-trained with real adult data, and the domain confusion component is pre-trained with real adult data and synthetic adult data.
15 . The system of claim 2 , wherein the one or more imaging devices are selected from a video camera, a motion capture device, a red-green-blue (RGB) camera, a long-wavelength infrared (LWIR) imaging device, and a depth sensor.
16 . A method of estimating a pose of an infant comprising:
receiving one or more images of an infant transmitted from an imaging device; and
determining a pose or posture of the infant by processing the one or more images with a pose estimator, wherein the pose estimator includes a trained model for estimating infant poses trained on an adult pose dataset and an augmented dataset containing real infant pose data and synthetic infant pose data using domain adaptation to align features of the synthetic infant pose data with the real infant pose data.
17 . The method of claim 16 , wherein the
one or more images of the infant are captured using the imaging device.
18 . The method of claim 16 , wherein the augmented dataset is generated by:
fitting infant model pose and shape parameters into a real infant image to form a reposed model, the infant model pose and shape parameters including body joints and shape coefficients, the shape coefficients including representations of height, length, fatness, thinness, and head-to-body ratio; and
generating synthetic output images, wherein the synthetic output images are generated by an imaging process as a function of the pose and shape parameters, camera parameters, body texture maps, and background images, the camera parameters including camera principal point and focal length, and the body texture maps including infant textures from infant clothing images augmented with adult textures from adult clothing images.
19 . The method of claim 18 , wherein the synthetic output images are augmented by minimizing a cost function, wherein the cost function is a sum of loss terms weighted by camera parameters, the loss terms including:
a joint-based data term comprising a distance between ground truth two dimensional joints and a two dimensional projection of corresponding posed three dimensional joints of the infant model pose for each joint;
a mixture of Gaussian pose priors learned from adult poses;
a shape penalty comprising a distance between a shape prior of the infant model and the shape parameters being optimized; and
a pose prior penalizing elbows and knees.
20 . The method of claim 16 , wherein:
the pose estimator comprises:
a pose estimation component including a feature extractor and a pose predictor; and
a domain confusion component in communication with the pose estimation component to share the feature extractor and including a domain classifier operative to distinguish whether a feature on an input image belongs to a real image or synthetic image; and
wherein the method further comprises:
training the pose estimator in a first stage, wherein the domain confusion component is fine-tuned using real infant pose data and synthetic infant pose data, to obtain a domain classifier for predicting whether features of an input image are from a synthetic infant image or a real infant image, using an optimization of a loss function to determine a binary cross entropy loss;
training the pose estimator in a second stage, wherein the pose estimation component is fine-tuned using the domain classifier to extract body keypoint information independently of differences between a real domain and a synthetic domain; and
balancing weights of real data and synthetic data in the second stage by maximizing a domain loss function, wherein features representing the real domain and features representing the synthetic domain become more similar.
21 . The method of claim 20 , further comprising pre-training the pose estimation component with real adult data, and pre-training the domain confusion component with real adult data and synthetic adult data.
22 . The system of claim 3 , wherein the shape coefficients include representations of height, length, fatness, thinness, and head-to-body ratio.