Automated costume augmentation using shape estimation

1. An automated costume augmentation system comprising a computing platform including a hardware processor and a system memory storing a software code, the hardware processor configured to execute the software code to:

determine a three-dimensional (3D) pose corresponding to a two-dimensional (2D) skeleton data including a plurality of joint positions corresponding to a posed figure in an image, using a first optimization algorithm applied to the 2D skeleton data, the first optimization algorithm including a first objective function;

identify at least one proportion of the posed figure based on the 2D skeleton data;

determine a plurality of bone directions corresponding to the posed figure, using a second optimization algorithm applied to the 3D pose, wherein the second optimization algorithm includes the first objective function summed with a second objective function configured to match the plurality of bone directions;

parameterize a costume for fitting to the posed figure based on the 3D pose, the at least one proportion, and the plurality of bone directions; and

output an enhanced image including the posed figure augmented with the costume fitted to the posed figure for rendering on a display.

2. The automated costume augmentation system of claim 1 , further comprising a 3D poses library accessible by the software code, wherein the 3D pose corresponding to the posed figure is determined using the first optimization algorithm applied to the 2D skeleton data and a plurality of 3D poses stored in the 3D poses library.

3. The automated costume augmentation system of claim 1 , wherein the at least one proportion of the posed figure includes at least one of a shoulder-width to hip-width ratio or a shoulder-width to average upper body height ratio of the posed figure.

4. The automated costume augmentation system of claim 1 , wherein, after parameterizing the costume for fitting to the posed figure and before outputting the enhanced image, the hardware processor is further configured to execute the software code to:

cover a body portion of the posed figure and an adjacent background portion of the image with a mask, wherein at least a head of the posed figure is not covered by the mask;

inpaint the mask to produce an inpainted mask having a restored background portion of the image; and

overlay the inpainted mask with the costume to produce the enhanced image.

5. The automated costume augmentation system of claim 1 , wherein the image comprises a single monocular image.

6. The automated costume augmentation system of claim 1 , wherein the computing platform is part of a personal communication device remote from an artificial neural network (ANN) generating the 2D skeleton data, the personal communication device further comprising the display and a camera.

7. The automated costume augmentation system of claim 6 , wherein the hardware processor is further configured to execute the software code to obtain the image using the camera.

8. The automated costume augmentation system of claim 6 , wherein the hardware processor is further configured to execute the software code to render the enhanced image on the display.

9. A method for use by an automated costume augmentation system including a computing platform having a hardware processor executing a software code stored in a system memory, the method comprising:

determining a three-dimensional (3D) pose corresponding to a two-dimensional (2D) skeleton data including a plurality of joint positions corresponding to a posed figure in an image, using a first optimization algorithm applied to the 2D skeleton data, the first optimization algorithm including a first objective function;

identifying at least one proportion of the posed figure based on the 2D skeleton data;

determining a plurality of bone directions corresponding to the posed figure, using a second optimization algorithm applied to the 3D pose, wherein the second optimization algorithm includes the first objective function summed with a second objective function configured to match the plurality of bone directions;

parameterizing a costume for fitting to the posed figure based on the 3D pose, the at least one proportion, and the plurality of bone directions; and

outputting an enhanced image including the posed figure augmented with the costume fitted to the posed figure for rendering on a display.

10. The method of claim 9 , wherein the automated costume augmentation system further comprises a 3D poses library accessible by the software code, wherein determining the 3D pose corresponding to the posed figure comprises applying the first optimization algorithm to the 2D skeleton data and a plurality of 3D poses stored in the 3D poses library.

11. The method of claim 9 , wherein the at least one proportion of the posed figure includes at least one of a shoulder-width to hip-width ratio or a shoulder-width to average upper body height ratio of the posed figure.

12. The method of claim 9 , wherein the method further comprises, after parameterizing the costume for fitting to the posed figure and before outputting the enhanced image:

covering, by the software code executed by the hardware processor, a body portion of the posed figure and an adjacent background portion of the image with a mask, wherein at least a head of the posed figure is not covered by the mask;

inpainting, by the software code executed by the hardware processor, the mask to produce an inpainted mask having a restored background portion of the image; and

overlaying, by the software code executed by the hardware processor, the inpainted mask with the costume to produce the enhanced image.

13. The method of claim 9 , wherein the image comprises a single monocular image.

14. The method of claim 9 , wherein the computing platform is part of a personal communication device remote from an artificial neural network (ANN) generating the 2D skeleton data, the personal communication device further comprising the display and a camera.

15. The method of claim 14 , further comprising obtaining the image using the camera.

16. The method of claim 14 , further comprising rendering the enhanced image on the display.

17. A computer-readable non-transitory medium having stored thereon a software code including instructions, which when executed by a hardware processor, instantiate a method comprising:

determining a three-dimensional (3D) pose corresponding to a two-dimensional (2D) skeleton data including a plurality of joint positions corresponding to a posed figure in an image, using a first optimization algorithm applied to the 2D skeleton data, the first optimization algorithm including a first objective function;

identifying at least one proportion of the posed figure based on the 2D skeleton data;

determining a plurality of bone directions corresponding to the posed figure, using a second optimization algorithm applied to the 3D pose, wherein the second optimization algorithm includes the first objective function summed with a second objective function configured to match the plurality of bone directions;

parameterizing a costume for fitting to the posed figure based on the 3D pose, the at least one proportion, and the plurality of bone directions; and

outputting an enhanced image including the posed figure augmented with the costume fitted to the posed figure for rendering on a display.

18. The computer-readable non-transitory medium of claim 17 , wherein determining the 3D pose corresponding to the posed figure comprises applying the first optimization algorithm to the 2D skeleton data and a plurality of 3D poses stored in a 3D poses library accessible by the software code.

19. The computer-readable non-transitory medium of claim 17 , wherein the at least one proportion of the posed figure includes at least one of a shoulder-width to hip-width ratio or a shoulder-width to average upper body height ratio of the posed figure.

20. The computer-readable non-transitory medium of claim 17 , wherein the method further comprises, after parameterizing the costume for fitting to the posed figure and before outputting the enhanced image:

covering a body portion of the posed figure and an adjacent background portion of the image with a mask, wherein at least a head of the posed figure is not covered by the mask;

inpainting the mask to produce an inpainted mask having a restored background portion of the image; and

overlaying the inpainted mask with the costume to produce the enhanced image.