Method and system for obtaining human body size information from image data
A computer implemented method of obtaining measurements of a person is disclosed herein. The method comprises obtaining a parametric deformable three-dimensional body model that can approximate the shape of any person, obtaining at least one image of the person, estimating one or more correspondences between the at least one image and the three-dimensional model, performing semantic image segmentation to segment the image of the person into their corresponding body parts, and iteratively adjusting at least one of (a) body pose and (b) shape parameters of the parametric deformable three-dimensional body model, to improve the fit of the three-dimensional model to at least one of: (i) the at least one image, (ii) the estimated one or more correspondences, and (iii) the segmented body parts. Measurements may then be extracted from the iteratively adjusted parametric deformable three-dimensional body model.
1 . A computer implemented method of obtaining measurements of a person, the method comprising: obtaining a parametric deformable three-dimensional body model that can approximate a shape of any person; obtaining at least one image of the person; estimating one or more correspondences between the at least one image and the parametric deformable three-dimensional body model; performing semantic image segmentation to segment the at least one image of the person into their corresponding body parts; iteratively adjusting at least one of (a) body pose and (b) shape parameters of the parametric deformable three-dimensional body model, to improve a fit of the parametric deformable three-dimensional body model to at least one of: (i) the at least one image, (ii) the estimated one or more correspondences; and (iii) the segmented body parts; and extracting measurements from the iteratively adjusted parametric deformable three-dimensional body model; wherein iteratively adjusting at least one of body pose and shape parameters of the parametric deformable three-dimensional body model, until the parametric deformable three-dimensional body model is consistent with (i) the at least one image, (ii) the estimated one or more correspondences, and (iii) the segmented body parts comprises: determining a reprojection error between a two-dimensional location and a projection of a corresponding three-dimensional point in the parametric deformable three-dimensional body model; and iteratively adjusting at least one of body pose and shape parameters until the reprojection error is less than a selected threshold; wherein the method further comprises determining a difference between segments given by projecting a semantically labelled current model estimate into each image and comparing with the segmented image of the person; wherein the method further comprises determining a two-dimensional chamfer distance between (i) pixel positions with a given semantic label and (ii) projections of three-dimensional vertices with a same semantic label from a current estimate of the parametric deformable three-dimensional body model, and wherein segmentation errors are weighted so that pixels lying near a boundary of the segments are given higher weight.
2 . The method of claim 1 , wherein estimating one or more correspondences between the at least one image and the parametric deformable three-dimensional body model comprises obtaining a correspondence between a two-dimensional location in the at least one image and an index to either (i) a point on a surface or (ii) an internal joint of the parametric deformable three-dimensional body model.
3 . The method of claim 1 , further comprising determining photometric consistency between images by using a current value of the parameters of the parametric deformable three-dimensional body model to cross-project colour values between pairs of images and obtain an error based on a difference between colour values from two different images of a same model.
4 . The method of claim 3 , wherein the photometric consistency error is evaluated at each mesh vertex, and differentiable bilinear sampling is used to sample image colours onto mesh vertices, wherein the sampling for each image is restricted to those vertices that are visible for that image.
5 . The method of claim 1 , further comprising determining photometric consistency between features derived from the at least one image extracted from a first few layers of a pretrained convolutional neural network.
6 . The method of claim 1 , wherein iteratively adjusting at least one of body pose and shape parameters of the parametric deformable three-dimensional body model, until the parametric deformable three-dimensional body model is consistent with (i) the at least one image, (ii) the estimated one or more correspondences, and (iii) the segmented body parts comprises:
obtaining a weighted sum of at least one of correspondence, segment and image consistency objective functions; and
minimising the weighted sum using a gradient-based nonlinear optimisation method.
7 . The method of claim 1 , wherein estimating one or more correspondences between the at least one image and the parametric deformable three-dimensional body model comprises obtaining a confidence value for each estimated correspondence indicating how confident a system is that the correspondence is correct, wherein the reprojection error for each correspondence is weighted by a confidence in a detection.
8 . The method of claim 1 , wherein estimating one or more correspondences comprises estimating a correspondence between a plurality of locations in the at least one image and a plurality of features of the parametric deformable three-dimensional body model.
9 . The method of claim 1 , further comprising instructing a user to assume a selected pose, and wherein an initial set of body pose parameters in an iterative optimisation process are selected to give a body in a same approximate pose that the user was instructed to assume.
10 . The method of claim 1 , further comprising iteratively adjusting intrinsic and extrinsic camera parameters, wherein initial values for the intrinsic camera parameters are obtained by looking up a camera model, based on image meta data associated with the at least one image of the person, in a database, and wherein initial values for the extrinsic camera parameters are obtained by solving a perspective-n-point, PnP, problem for each image using the estimated correspondences between the at least one image and the parametric deformable three-dimensional body model, and wherein three-dimensional coordinates for the PnP problem are obtained by a corresponding surface or joint positions on a body model having an initial set of shape parameters as determined based on a reference body shape and body pose parameters selected to give a body in a same approximate pose that a user was instructed to assume, further comprising obtaining accelerometer information associated with each at least one image of the person, and wherein the initial values for the extrinsic parameters are obtained by solving a restricted PnP problem for each image in which a rotation is constrained based on the accelerometer information.
11 . The method of claim 1 , wherein the method further comprises obtaining a user-supplied height, and wherein obtaining a parametric deformable three-dimensional body model of a person is based on the obtained user-supplied height such that the parametric deformable three-dimensional body model of a person has a height equal to the obtained user-supplied height.
12 . The method of claim 11 , wherein iteratively adjusting at least one of body pose and shape parameters of the parametric deformable three-dimensional body model, until the parametric deformable three-dimensional body model is consistent with (i) the at least one image, (ii) the estimated one or more correspondences, and (iii) the segmented body parts comprises using an objective function that is augmented by a term that penalises any difference between the height of the parametric deformable three-dimensional body model and the user-supplied height.
13 . The method of claim 1 , wherein a plurality of images of the person are obtained, and the method comprises iteratively adjusting at least one of body pose and shape parameters of the parametric deformable three-dimensional body model, until the parametric deformable three-dimensional body model is consistent with (i) the at least one image, (ii) the estimated one or more correspondences, and (iii) the segmented body parts comprises using an objective function that is augmented by a term that penalises a least one of (a) variation in a body-to-camera translation between images, and (b) variation in the body pose between images.
14 . The method of claim 1 , wherein the method comprises iteratively adjusting at least one of body pose and shape parameters of the parametric deformable three-dimensional body model, until the parametric deformable three-dimensional body model is consistent with (i) the at least one image, (ii) the estimated one or more correspondences, and (iii) the segmented body parts comprises using an objective function that is augmented by a term that penalises at least one of (a) implausible shapes using a known prior distribution of the shape parameters in the parametric deformable three-dimensional body model, and (b) implausible body poses.
15 . A computer readable non-transitory storage medium comprising a program for a computer configured to cause a processor to perform a method of obtaining measurements of a person, the method comprising: obtaining a parametric deformable three-dimensional body model that can approximate a shape of any person; obtaining at least one image of the person; estimating one or more correspondences between the at least one image and the parametric deformable three-dimensional body model; performing semantic image segmentation to segment the at least one image of the person into their corresponding body parts; iteratively adjusting at least one of (a) body pose and (b) shape parameters of the parametric deformable three-dimensional body model, to improve a fit of the parametric deformable three-dimensional body model to at least one of: (i) the at least one image, (ii) the estimated one or more correspondences; and (iii) the segmented body parts; and extracting measurements from the iteratively adjusted parametric deformable three-dimensional body model; wherein iteratively adjusting at least one of body pose and shape parameters of the parametric deformable three-dimensional body model, until the parametric deformable three-dimensional body model is consistent with (i) the at least one image, (ii) the estimated one or more correspondences, and (iii) the segmented body parts comprises: determining a reprojection error between a two-dimensional location and a projection of a corresponding three-dimensional point in the parametric deformable three-dimensional body model; and iteratively adjusting at least one of body pose and shape parameters until the reprojection error is less than a selected threshold; wherein the method further comprises determining a difference between segments given by projecting a semantically labelled current model estimate into each image and comparing with the segmented image of the person; wherein the method further comprises determining a two-dimensional chamfer distance between (i) pixel positions with a given semantic label and (ii) projections of three-dimensional vertices with a same semantic label from a current estimate of the parametric deformable three-dimensional body model, and wherein segmentation errors are weighted so that pixels lying near a boundary of the segments are given higher weight.
16 . A system comprising: a processor; and a computer readable non-transitory storage medium comprising a program for a computer configured to cause the processor to perform a method of obtaining measurements of a person, the method comprising: obtaining a parametric deformable three-dimensional body model that can approximate a shape of any person; obtaining at least one image of the person; estimating one or more correspondences between the at least one image and the parametric deformable three-dimensional body model; performing semantic image segmentation to segment the at least one image of the person into their corresponding body parts; iteratively adjusting at least one of (a) body pose and (b) shape parameters of the parametric deformable three-dimensional body model, to improve a fit of the parametric deformable three-dimensional body model to at least one of: (i) the at least one image, (ii) the estimated one or more correspondences; and (iii) the segmented body parts; and extracting measurements from the iteratively adjusted parametric deformable three-dimensional body model; wherein iteratively adjusting at least one of body pose and shape parameters of the parametric deformable three-dimensional body model, until the parametric deformable three-dimensional body model is consistent with (i) the at least one image, (ii) the estimated one or more correspondences, and (iii) the segmented body parts comprises: determining a reprojection error between a two-dimensional location and a projection of a corresponding three-dimensional point in the parametric deformable three-dimensional body model; and iteratively adjusting at least one of body pose and shape parameters until the reprojection error is less than a selected threshold; wherein the method further comprises determining a difference between segments given by projecting a semantically labelled current model estimate into each image and comparing with the segmented image of the person; wherein the method further comprises determining a two-dimensional chamfer distance between (i) pixel positions with a given semantic label and (ii) projections of three-dimensional vertices with a same semantic label from a current estimate of the parametric deformable three-dimensional body model, and wherein segmentation errors are weighted so that pixels lying near a boundary of the segments are given higher weight.