Method and apparatus for stylizing three-dimensional model, electronic device, and storage medium

1 . A method for stylizing a three-dimensional model, comprising:

acquiring a to-be-stylized three-dimensional model and a stylized target image; and

rendering the three-dimensional model by using a predetermined network to acquire a two-dimensional rendered image and spatial feature parameters of a pixel, and stylizing a texture feature in the two-dimensional rendered image based on the spatial feature parameters and the stylized target image to acquire a stylized three-dimensional model, the stylizing comprising:

in response to determining a plurality of viewing angles whose field-of-view ranges cover the three-dimensional model and the field-of-view ranges of adjacent viewing angles are overlapped, stylizing the texture feature of a portion that is in the two-dimensional rendered image corresponding to a current viewing angle of the plurality of viewing angles but does not appear in a two-dimensional rendered image corresponding to a previous viewing angle of the plurality of viewing angles, wherein the previous viewing angle is a viewing angle from which the rendering operation and the stylizing operation have been performed.

2 . The method according to claim 1 , wherein the acquiring a to-be-stylized three-dimensional model and a stylized target image comprises:

scanning at least two two-dimensional input images, wherein each two-dimensional input image comprises a feature of a to-be-modeled target in a corresponding viewing angle; and

establishing the three-dimensional model of the to-be-modeled target based on the at least two two-dimensional input images.

3 . The method according to claim 1 , wherein the rendering the three-dimensional model by using the predetermined network to acquire the two-dimensional rendered image and the spatial feature parameters of the pixel, and stylizing the texture feature in the two-dimensional rendered image based on the spatial feature parameters and the stylized target image to acquire the stylized three-dimensional model comprises:

determining the plurality of viewing angles;

rendering the three-dimensional model from the current viewing angle by using the predetermined network to acquire a two-dimensional rendered image corresponding to the current viewing angle and spatial feature parameters of a pixel corresponding to the current viewing angle, and stylizing the texture feature in the two-dimensional rendered image corresponding to the current viewing angle based on the spatial feature parameters of the pixel corresponding to the current viewing angle and the stylized target image; and

continuing to select a next viewing angle as a current viewing angle, and repeatedly performing a rendering operation and a stylizing operation from the current viewing angle until a complete stylized three-dimensional model is acquired.

4 . The method according to claim 3 , wherein the predetermined network comprises a renderer and a stylizing network;

the spatial feature parameters of the pixel corresponding to the current viewing angle comprise an included angle between a normal direction and a sight line that correspond to the pixel corresponding to the current viewing angle, as well as a depth of the pixel corresponding to the current viewing angle; and

in a case that the current viewing angle is not a first viewing angle, the spatial feature parameters of the pixel corresponding to the current viewing angle further comprise a mask of a portion that is in the two-dimensional rendered image corresponding to the current viewing angle and is already stylized in the previous viewing angle.

5 . The method according to claim 3 , further comprising:

training the predetermined network based on a sample model and the stylized target image until a value of a loss function of the predetermined network satisfies a requirement, wherein the value of the loss function is determined based on values of the following three functions:

a content loss function configured to evaluate a loss between a stylized result and the sample model;

a style loss function configured to evaluate a loss between the stylized result and the stylized target image; and

a continuity loss function configured to evaluate a super-resolution test sequence VGGloss between the stylized result and a sample model partially stylized in the previous viewing angle.

6 . The method according to claim 5 , wherein the style loss function is a weighted sum of losses between an image feature of the stylized target image and image features of a plurality of pixels in the stylized result, wherein a weight corresponding to each pixel is negatively correlated with an included angle between the normal direction and a sight line that correspond to the pixel.

7 . The method according to claim 6 , wherein an image feature of each pixel in the stylized result is determined by performing weighted convolution on pixels surrounding the pixel, wherein a weight of the convolution is negatively correlated with a depth difference between the pixel and the surrounding pixels.

8 . The method according to claim 5 , wherein the continuity loss function is a weighted sum of VGG losses between a pixel corresponding to the sample model partially stylized in the previous viewing angle and a plurality of pixels in the stylized result, wherein a weight corresponding to each pixel is positively correlated with an included angle between the normal direction and a sight line that correspond to the pixel.

9 . An electronic device, comprising:

at least one processor; and

a storage device, configured to store at least one program, wherein the at least one program, when executed by the at least one processor, causes the at least one processor to implement acts comprising:

acquiring a to-be-stylized three-dimensional model and a stylized target image; and

rendering the three-dimensional model by using a predetermined network to acquire a two-dimensional rendered image and spatial feature parameters of a pixel, and stylizing a texture feature in the two-dimensional rendered image based on the spatial feature parameters and the stylized target image to acquire a stylized three-dimensional model, the stylizing comprising:

in response to determining a plurality of viewing angles whose field-of-view ranges cover the three-dimensional model and the field-of-view ranges of adjacent viewing angles are overlapped, stylizing the texture feature of a portion that is in the two-dimensional rendered image corresponding to a current viewing angle of the plurality of viewing angles but does not appear in a two-dimensional rendered image corresponding to a previous viewing angle of the plurality of viewing angles, wherein the previous viewing angle is a viewing angle from which the rendering operation and the stylizing operation have been performed.

10 . The electronic device according to claim 9 , wherein the acquiring a to-be-stylized three-dimensional model and a stylized target image comprises:

scanning at least two two-dimensional input images, wherein each two-dimensional input image comprises a feature of a to-be-modeled target in a corresponding viewing angle; and

establishing the three-dimensional model of the to-be-modeled target based on the at least two two-dimensional input images.

11 . The electronic device according to claim 9 , wherein the rendering the three-dimensional model by using the predetermined network to acquire the two-dimensional rendered image and the spatial feature parameters of the pixel, and stylizing the texture feature in the two-dimensional rendered image based on the spatial feature parameters and the stylized target image to acquire the stylized three-dimensional model comprises:

determining the plurality of viewing angles;

rendering the three-dimensional model from the current viewing angle by using the predetermined network to acquire a two-dimensional rendered image corresponding to the current viewing angle and spatial feature parameters of a pixel corresponding to the current viewing angle, and stylizing the texture feature in the two-dimensional rendered image corresponding to the current viewing angle based on the spatial feature parameters of the pixel corresponding to the current viewing angle and the stylized target image; and

continuing to select a next viewing angle as a current viewing angle, and repeatedly performing a rendering operation and a stylizing operation from the current viewing angle until a complete stylized three-dimensional model is acquired.

12 . The electronic device according to claim 11 , wherein the predetermined network comprises a renderer and a stylizing network;

the spatial feature parameters of the pixel corresponding to the current viewing angle comprise an included angle between a normal direction and a sight line that correspond to the pixel corresponding to the current viewing angle, as well as a depth of the pixel corresponding to the current viewing angle; and

in a case that the current viewing angle is not a first viewing angle, the spatial feature parameters of the pixel corresponding to the current viewing angle further comprise a mask of a portion that is in the two-dimensional rendered image corresponding to the current viewing angle and is already stylized in the previous viewing angle.

13 . The electronic device according to claim 11 , the acts further comprising:

training the predetermined network based on a sample model and the stylized target image until a value of a loss function of the predetermined network satisfies a requirement, wherein the value of the loss function is determined based on values of the following three functions:

a content loss function configured to evaluate a loss between a stylized result and the sample model;

a style loss function configured to evaluate a loss between the stylized result and the stylized target image; and

a continuity loss function configured to evaluate a super-resolution test sequence VGG loss between the stylized result and a sample model partially stylized in the previous viewing angle.

14 . A non-transitory computer-readable storage medium, configured to store a computer program, wherein the program, when executed by a processor, causes the processor to implement acts comprising:

acquiring a to-be-stylized three-dimensional model and a stylized target image; and

rendering the three-dimensional model by using a predetermined network to acquire a two-dimensional rendered image and spatial feature parameters of a pixel, and stylizing a texture feature in the two-dimensional rendered image based on the spatial feature parameters and the stylized target image to acquire a stylized three-dimensional model, the stylizing comprising:

in response to determining a plurality of viewing angles whose field-of-view ranges cover the three-dimensional model and the field-of-view ranges of adjacent viewing angles are overlapped, stylizing the texture feature of a portion that is in the two-dimensional rendered image corresponding to a current viewing angle of the plurality of viewing angles but does not appear in a two-dimensional rendered image corresponding to a previous viewing angle of the plurality of viewing angles, wherein the previous viewing angle is a viewing angle from which the rendering operation and the stylizing operation have been performed.

15 . The non-transitory computer-readable storage medium according to claim 14 , wherein the acquiring a to-be-stylized three-dimensional model and a stylized target image comprises:

scanning at least two two-dimensional input images, wherein each two-dimensional input image comprises a feature of a to-be-modeled target in a corresponding viewing angle; and

establishing the three-dimensional model of the to-be-modeled target based on the at least two two-dimensional input images.

16 . The non-transitory computer-readable storage medium according to claim 14 , wherein the rendering the three-dimensional model by using the predetermined network to acquire the two-dimensional rendered image and the spatial feature parameters of the pixel, and stylizing the texture feature in the two-dimensional rendered image based on the spatial feature parameters and the stylized target image to acquire the stylized three-dimensional model comprises:

determining the plurality of viewing angles;

rendering the three-dimensional model from the current viewing angle by using the predetermined network to acquire a two-dimensional rendered image corresponding to the current viewing angle and spatial feature parameters of a pixel corresponding to the current viewing angle, and stylizing the texture feature in the two-dimensional rendered image corresponding to the current viewing angle based on the spatial feature parameters of the pixel corresponding to the current viewing angle and the stylized target image; and

continuing to select a next viewing angle as a current viewing angle, and repeatedly performing a rendering operation and a stylizing operation from the current viewing angle until a complete stylized three-dimensional model is acquired.

17 . The non-transitory computer-readable storage medium according to claim 16 , wherein the predetermined network comprises a renderer and a stylizing network;

the spatial feature parameters of the pixel corresponding to the current viewing angle comprise an included angle between a normal direction and a sight line that correspond to the pixel corresponding to the current viewing angle, as well as a depth of the pixel corresponding to the current viewing angle; and

in a case that the current viewing angle is not a first viewing angle, the spatial feature parameters of the pixel corresponding to the current viewing angle further comprise a mask of a portion that is in the two-dimensional rendered image corresponding to the current viewing angle and is already stylized in the previous viewing angle.