Generating realistic makeup in a digital video stream
Methods, systems, and non-transitory computer readable storage media are disclosed for rendering makeup products on a user's face within an augmented reality environment in real-time. For example, the disclosed system can use blend a base makeup color of a selected makeup product with extracted luminance frequencies of a video stream. The disclosed system can then convert the blended makeup color to LAB color space and apply one or more shader models to the lightness of the color based on a material of the selected makeup product. The disclosed system can also apply additional operations for smoothing the skin of the user and matching the makeup product to a skin tone of the user. The disclosed system can then display the makeup product and any additional changes in an augmented reality environment in the video stream.
1. A computer-implemented method comprising:
receiving a makeup selection to apply to an area of an identified face in a digital video stream, the makeup selection comprising an albedo color in an RGB color space and a material including a texture characteristic, wherein the albedo color of the makeup selection comprises a base color of a selected makeup product in the RGB color space;
extracting luminance frequencies from the digital video stream;
blending, in the RGB color space, the albedo color of the makeup selection with the luminance frequencies of the digital video stream;
converting the blended color of the makeup selection to a LAB color space;
modifying, in the LAB color space, a lightness of the blended color by applying at least one shading model based on the texture characteristic of the material corresponding to the makeup selection; and
presenting, via a display screen of a client device, the makeup selection comprising the blended color with the modified lightness on the area of the identified face in the digital video stream.
2. The computer-implemented method as recited in claim 1 , further comprising converting the blended color with the modified lightness from the LAB color space to the RGB color space prior to presenting the makeup selection via the display screen of the client device.
3. The computer-implemented method as recited in claim 1 , wherein extracting the luminance frequencies from the digital video stream comprises:
normalizing luminance values of the digital video stream; and
extracting, from the normalized luminance values, a set of mid-range frequencies and a set of low-range frequencies from a desaturated digital image frame of the digital video stream.
4. The computer-implemented method as recited in claim 3 , wherein blending the albedo color of the makeup selection with the luminance frequencies comprises:
blending the albedo color of the makeup selection with the set of mid-range frequencies by utilizing a linear interpolation function to screen the mid-range frequencies over the albedo color of the makeup selection to generate a first blend output; and
blending the first blend output with the set of low-range frequencies by multiplying the low-range frequencies on top of the first blend output to generate a second blend output.
5. The computer-implemented method as recited in claim 4 , wherein converting the blended color of the makeup selection to the LAB color space comprises converting the second blend output to the LAB color space.
6. The computer-implemented method as recited in claim 1 , wherein presenting the makeup selection comprising the blended color with the modified lightness on the area of the identified face in the digital video stream comprises:
generating a three-dimensional mesh corresponding to the identified face; and
presenting the makeup selection on an area of the three-dimensional mesh corresponding to the area of the identified face.
7. The computer-implemented method as recited in claim 1 , further comprising:
receiving an accessory selection to attach an accessory to a second area of the identified face;
adding, in response to the accessory selection, an accessory mesh to a three-dimensional mesh corresponding to the identified face;
applying an accessory texture comprising the accessory to the accessory mesh; and
presenting, via the display screen of the client device, the accessory on the second area of the identified face in the digital video stream.
8. The computer-implemented method as recited in claim 7 , further comprising:
generating, in response to adding the accessory mesh to the three-dimensional mesh corresponding to the identified face, an animation armature comprising a plurality of joints for the accessory mesh; and
animating the accessory mesh using the animation armature in response to movement of the second area of the identified face in the digital video stream.
9. The computer-implemented method as recited in claim 1 , further comprising:
generating a guiding image based on a luma of the digital video stream; and
smoothing, using a fast guided blur filter, the identified face based on the guiding image.
10. The computer-implemented method as recited in claim 1 , further comprising:
dividing the digital video stream into a plurality of sections;
determining, for a section of the plurality of sections, an average log luma and a luma range; and
shifting, for the section of the plurality of sections, the average log luma based on a predetermined luma threshold to apply localized tone mapping to the digital video stream.
11. A non-transitory computer readable storage medium comprising instructions that, when executed by at least one processor, cause a computer system to:
receive a makeup selection to apply to an area of an identified face in a digital video stream, the makeup selection comprising an albedo color in an RGB color space and a material including a texture characteristic, wherein the albedo color of the makeup selection comprises a base color of a selected makeup product in the RGB color space;
extract luminance frequencies from the digital video stream;
blend, in the RGB color space, the albedo color of the makeup selection with the luminance frequencies of the digital video stream;
convert the blended color of the makeup selection into a LAB color space;
modify, in the LAB color space, a lightness of the blended color by applying at least one shading model based on the texture characteristic of the material corresponding to the makeup selection; and
present, via a display screen of a client device, the makeup selection comprising the blended color with the modified lightness on the area of the identified face in the digital video stream.
12. The non-transitory computer readable storage medium as recited in claim 11 , wherein the instructions further cause the computer system to:
extract a set of mid-range frequencies and a set of low-range frequencies from a desaturated digital image frame of the digital video stream;
blend the albedo color of the makeup selection with the set of mid-range frequencies by utilizing a linear interpolation function to screen the mid-range frequencies over the albedo color of the makeup selection to generate a first blend output; and
blend the first blend output with the set of low-range frequencies by multiplying the low-range frequencies on top of the first blend output to generate a second blend output.
13. The non-transitory computer readable storage medium as recited in claim 11 , further comprising instructions that, when executed by the at least one processor, cause the computer system to:
receive an accessory selection to attach an accessory to a second area of the identified face;
add, in response to the accessory selection, an accessory mesh to a three-dimensional mesh corresponding to the identified face;
apply an accessory texture comprising the accessory to the accessory mesh; and
present, via the display screen of the client device, the accessory on the second area of the identified face in the digital video stream.
14. The non-transitory computer readable storage medium as recited in claim 13 , further comprising instructions that, when executed by the at least one processor, cause the computer system to:
generate, in response to adding the accessory mesh to the three-dimensional mesh corresponding to the identified face, an animation armature comprising a plurality of joints for the accessory mesh; and
animate the accessory mesh using the animation armature in response to movement of the second area of the identified face in the digital video stream.
15. The non-transitory computer readable storage medium as recited in claim 11 , further comprising instructions that, when executed by the at least one processor, cause the computer system to:
generate a guiding image based on a luma of the digital video stream; and
smooth, using a fast guided blur filter, the identified face based on the guiding image.
16. The non-transitory computer readable storage medium as recited in claim 11 , further comprising instructions that, when executed by the at least one processor, cause the computer system to:
divide the digital video stream into a plurality of sections;
determine, for a section of the plurality of sections, an average log luma and a luma range; and
shift, for the section of the plurality of sections, the average log luma based on a predetermined luma threshold to apply localized tone mapping to the digital video stream.
17. The non-transitory computer readable storage medium as recited in claim 11 , further comprising instructions that, when executed by the at least one processor, cause the computer system to:
generating a downsampled version of the digital video stream; and
utilizing the downsampled version of the digital video stream to smooth the identified face and to apply localized tone mapping to the digital video stream.
18. A system comprising:
at least one processor; and
a non-transitory computer readable storage medium comprising instructions that, when executed by the at least one processor, cause the system to:
receive a makeup selection to apply to an area of an identified face in a digital video stream, the makeup selection comprising an albedo color in an RGB color space and a material including a texture characteristic, wherein the albedo color of the makeup selection comprises a base color of a selected makeup product in the RGB space;
extract luminance frequencies from the digital video stream;
blend, in the RGB color space, the albedo color of the makeup selection with the luminance frequencies of the digital video stream;
convert the blended color of the makeup selection into a LAB color space;
modify, in the LAB color space, a lightness of the blended color by applying at least one shading model based on the texture characteristic of the material corresponding to the makeup selection; and
present, via a display screen of a client device, the makeup selection comprising the blended color with the modified lightness on the area of the identified face in the digital video stream.
19. The system as recited in claim 18 , wherein the instructions further cause the system to:
extract a set of mid-range frequencies and a set of low-range frequencies from a desaturated digital image frame of the digital video stream;
blend the albedo color of the makeup selection with the set of mid-range frequencies by utilizing a linear interpolation function to screen the mid-range frequencies over the albedo color to generate a first blend output; and
blend the first blend output with the set of low-range frequencies by multiplying the low-range frequencies on top of the first blend output to generate a second blend output.
20. The system as recited in claim 18 , further comprising instructions that cause the system to:
smooth, using a fast guided filter, the identified face using a guiding image based on a luma of the digital video stream; and
shift an average log luma of a plurality of sections of the digital video stream based on a predetermined luma threshold to apply localized tone mapping to the digital video stream.