Image rendering method, electronic device, and storage medium
An image rendering method includes the steps below. A model of an environmental object is rendered to obtain an image of the environmental object in a target perspective. An image of a target object in the target perspective and a model of the target object are determined according to a neural radiance field of the target object. The image of the target object is fused and rendered into the image of the environmental object according to the model of the target object.
1 . An image rendering method, comprising:
rendering a model of an environmental object to obtain an image of the environmental object in a target perspective;
determining an image of a target object in the target perspective and a model of the target object according to a neural radiance field of the target object; and
fusing and rendering the image of the target object into the image of the environmental object according to the model of the target object;
wherein fusing and rendering the image of the target object into the image of the environmental object according to the model of the target object comprises: determining a normal direction of a target vertex in the model of the target object according to the model of the target object; performing a light adjustment on the image of the target object to obtain a light-adjusted image of the target object according to the target perspective, environmental light, and the normal direction of the target vertex; and fusing and rendering the light-adjusted image of the target object into the image of the environmental object; or
wherein fusing and rendering the image of the target object into the image of the environmental object according to the model of the target object comprises: determining depth information of a target vertex in the model of the target object according to the target perspective and the model of the target object; comparing depth information of a respective target vertex at each pixel in the model of the target object with depth information of a respective environmental object at the each pixel; in response to determining that the depth information of the respective target vertex is smaller than the depth information of the respective environmental object, determining that the respective target vertex is not occluded by the respective environmental object, and replacing a pixel value of the respective environmental object by a pixel value of the respective target vertex; and in response to determining that the depth information of the respective target vertex is greater than or equal to the depth information of the respective environmental object, determining that the respective target vertex is occluded by the respective environmental object, and discarding the pixel value of the respective target vertex;
wherein determining the image of the target object in the target perspective and the model of the target object according to the neural radiance field of the target object comprises: acquiring a point cloud model of the target object from the neural radiance field of the target object, and processing the point cloud model of the target object to obtain a mesh model of the target object; and according to the neural radiance field of the target object, determining the image of the target object in the target perspective by using a camera projection matrix and a model view matrix in the target perspective;
wherein according to the neural radiance field of the target object, determining the image of the target object in the target perspective by using the camera projection matrix and the model view matrix in the target perspective comprises: inputting the camera projection matrix and the model view matrix in the target perspective to the neural radiance field of the target object so that the image of the target object in the target perspective is obtained by a neural rendering; and
wherein the neural radiance field of the target object is determined by: acquiring, in an acquisition perspective, data of the target object to obtain a two-dimensional image of the target object and a three-dimensional point cloud of the target object using a camera and a laser radar, wherein the camera shoots around a center of visual field at different heights and in a case where the camera is rotated, the camera is rotated slowly and moved back and forth to increase coincidence rate of visual field between adjacent frames; fusing the two-dimensional image of the target object and the three-dimensional point cloud of the target object to obtain a fused image of the target object; and determining the neural radiance field of the target object according to the fused image of the target object and the acquisition perspective.
2 . The method according to claim 1 , wherein performing the light adjustment on the image of the target object to obtain the light-adjusted image of the target object according to the target perspective, the environmental light, and the normal direction of the target vertex comprises:
transforming, by using the model view matrix in the target perspective, the normal direction of the target vertex to obtain a transformed normal direction of the target vertex;
determining an intensity of the target vertex according to the environmental light and the transformed normal direction of the target vertex; and
performing the light adjustment on the image of the target object to obtain the light-adjusted image of the target object according to the intensity of the target vertex.
3 . The method according to claim 1 , wherein determining the depth information of the target vertex in the model of the target object according to the target perspective and the model of the target object comprises:
transforming, by using the model view matrix in the target perspective, the model of the target object to obtain a transformed model of the target object; and
projecting, by using a camera projection matrix in the target perspective, the transformed model of the target object to obtain the depth information of the target vertex in the model of the target object.
4 . The method according to claim 1 , wherein processing the point cloud model of the target object to obtain the mesh model of the target object comprises:
according to a marching cubes algorithm, processing the point cloud model of the target object to obtain the mesh model of the target object.
5 . An electronic device, comprising:
at least one processor; and
a memory communicatively connected to the at least one processor,
wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to cause the at least one processor to execute:
rendering a model of an environmental object to obtain an image of the environmental object in a target perspective;
determining an image of a target object in the target perspective and a model of the target object according to a neural radiance field of the target object; and
fusing and rendering the image of the target object into the image of the environmental object according to the model of the target object;
wherein the at least one processor fuses and renders the image of the target object into the image of the environmental object according to the model of the target object by: determining a normal direction of a target vertex in the model of the target object according to the model of the target object; performing a light adjustment on the image of the target object to obtain a light-adjusted image of the target object according to the target perspective, environmental light, and the normal direction of the target vertex; and fusing and rendering the light-adjusted image of the target object into the image of the environmental object; or
wherein the at least one processor fuses and renders the image of the target object into the image of the environmental object according to the model of the target object by: determining depth information of a target vertex in the model of the target object according to the target perspective and the model of the target object; comparing depth information of a respective target vertex at each pixel in the model of the target object with depth information of a respective environmental object at the each pixel; in response to determining that the depth information of the respective target vertex is smaller than the depth information of the respective environmental object, determining that the respective target vertex is not occluded by the respective environmental object, and replacing a pixel value of the respective environmental object by a pixel value of the respective target vertex; and in response to determining that the depth information of the respective target vertex is greater than or equal to the depth information of the respective environmental object, determining that the respective target vertex is occluded by the respective environmental object, and discarding the pixel value of the respective target vertex;
wherein determining the image of the target object in the target perspective and the model of the target object according to the neural radiance field of the target object comprises: acquiring a point cloud model of the target object from the neural radiance field of the target object, and processing the point cloud model of the target object to obtain a mesh model of the target object; and according to the neural radiance field of the target object, determining the image of the target object in the target perspective by using a camera projection matrix and a model view matrix in the target perspective;
wherein according to the neural radiance field of the target object, determining the image of the target object in the target perspective by using the camera projection matrix and the model view matrix in the target perspective comprises: inputting the camera projection matrix and the model view matrix in the target perspective to the neural radiance field of the target object so that the image of the target object in the target perspective is obtained by a neural rendering; and
wherein the neural radiance field of the target object is determined by: acquiring, in an acquisition perspective, data of the target object to obtain a two-dimensional image of the target object and a three-dimensional point cloud of the target object using a camera and a laser radar, wherein the camera shoots around a center of visual field at different heights and in a case where the camera is rotated, the camera is rotated slowly and moved back and forth to increase coincidence rate of visual field between adjacent frames; fusing the two-dimensional image of the target object and the three-dimensional point cloud of the target object to obtain a fused image of the target object; and determining the neural radiance field of the target object according to the fused image of the target object and the acquisition perspective.
6 . The electronic device according to claim 5 , wherein the at least one processor performs the light adjustment on the image of the target object to obtain the light-adjusted image of the target object according to the target perspective, the environmental light, and the normal direction of the target vertex by:
transforming, by using the model view matrix in the target perspective, the normal direction of the target vertex to obtain a transformed normal direction of the target vertex;
determining an intensity of the target vertex according to the environmental light and the transformed normal direction of the target vertex; and
performing the light adjustment on the image of the target object to obtain the light-adjusted image of the target object according to the intensity of the target vertex.
7 . The electronic device according to claim 5 , wherein the at least one processor determines the depth information of the target vertex in the model of the target object according to the target perspective and the model of the target object by:
transforming, by using the model view matrix in the target perspective, the model of the target object to obtain a transformed model of the target object; and
projecting, by using a camera projection matrix in the target perspective, the transformed model of the target object to obtain the depth information of the target vertex in the model of the target object.
8 . The electronic device according to claim 5 , wherein the at least one processor processes the point cloud model of the target object to obtain the mesh model of the target object by:
according to a marching cubes algorithm, processing the point cloud model of the target object to obtain the mesh model of the target object.
9 . A non-transitory computer-readable storage medium storing computer instructions for causing a computer to execute:
rendering a model of an environmental object to obtain an image of the environmental object in a target perspective;
determining an image of a target object in the target perspective and a model of the target object according to a neural radiance field of the target object; and
fusing and rendering the image of the target object into the image of the environmental object according to the model of the target object;
wherein fusing and rendering the image of the target object into the image of the environmental object according to the model of the target object comprises: determining a normal direction of a target vertex in the model of the target object according to the model of the target object; performing a light adjustment on the image of the target object to obtain a light-adjusted image of the target object according to the target perspective, environmental light, and the normal direction of the target vertex; and fusing and rendering the light-adjusted image of the target object into the image of the environmental object; or wherein fusing and rendering the image of the target object into the image of the environmental object according to the model of the target object comprises: determining depth information of a target vertex in the model of the target object according to the target perspective and the model of the target object; comparing depth information of a respective target vertex at each pixel in the model of the target object with depth information of a respective environmental object at the each pixel; in response to determining that the depth information of the respective target vertex is smaller than the depth information of the respective environmental object, determining that the respective target vertex is not occluded by the respective environmental object, and replacing a pixel value of the respective environmental object by a pixel value of the respective target vertex; and in response to determining that the depth information of the respective target vertex is greater than or equal to the depth information of the respective environmental object, determining that the respective target vertex is occluded by the respective environmental object, and discarding the pixel value of the respective target vertex; and
wherein determining the image of the target object in the target perspective and the model of the target object according to the neural radiance field of the target object comprises: acquiring a point cloud model of the target object from the neural radiance field of the target object, and processing the point cloud model of the target object to obtain a mesh model of the target object; and according to the neural radiance field of the target object, determining the image of the target object in the target perspective by using a camera projection matrix and a model view matrix in the target perspective;
wherein according to the neural radiance field of the target object, determining the image of the target object in the target perspective by using the camera projection matrix and the model view matrix in the target perspective comprises: inputting the camera projection matrix and the model view matrix in the target perspective to the neural radiance field of the target object so that the image of the target object in the target perspective is obtained by a neural rendering; and
wherein the neural radiance field of the target object is determined by: acquiring, in an acquisition perspective, data of the target object to obtain a two-dimensional image of the target object and a three-dimensional point cloud of the target object using a camera and a laser radar, wherein the camera shoots around a center of visual field at different heights and in a case where the camera is rotated, the camera is rotated slowly and moved back and forth to increase coincidence rate of visual field between adjacent frames; fusing the two-dimensional image of the target object and the three-dimensional point cloud of the target object to obtain a fused image of the target object; and determining the neural radiance field of the target object according to the fused image of the target object and the acquisition perspective.