Device and method with scene view image generation
An electronic device includes: a camera sensor, disposed on a body, and configured to generate an image including image data by sequentially scanning scan lines; and memory storing instructions configured to cause a processor to: obtain a default mesh with vertices corresponding to positions of image pixels along a scan line of the camera sensor and back-projected onto a target plane, obtain a planar mesh from the default mesh according to motion of the body, and generate a scene view image from the image data based on the obtained planar mesh.
1. An electronic device comprising:
a camera sensor, disposed on a body, and configured to generate an image comprising image data by sequentially scanning scan lines; and
memory storing instructions configured to cause a processor to:
obtain a default mesh with vertices corresponding to positions of image pixels along a scan line of the camera sensor and back-projected onto a target plane, the target plane being located and included in a field of view (FOV) of the camera sensor,
obtain a planar mesh from the default mesh according to motion of the body, and
generate a scene view image from the image data based on the obtained planar mesh.
2. The electronic device of claim 1 , wherein the instructions are further configured to cause the processor to:
obtain the planar mesh by moving a group of vertices of the default mesh that correspond to a same scan line captured at a scan time point, wherein the moving is based on motion of the body corresponding to the scan time point.
3. The electronic device of claim 1 , wherein the instructions are further configured to cause the processor to:
obtain motions of the body, the motions of the body respectively corresponding to scan time points when lines of the image data were captured by respective scan lines of the camera sensor;
obtain the planar mesh by:
for each time point, moving vertices associated with a corresponding scan line according to a displacement that is based on a difference between the corresponding motion of the body at the scan time point and a motion of the body at a reference time point.
4. The electronic device of claim 1 , wherein the instructions are further configured to cause the processor to:
when the body is moving in a straight line, obtain the planar mesh by moving vertices, in the default mesh, and corresponding to a scan line scanned at a scan time point subsequent to the reference time point, in a direction corresponding to the straight line movement of the body.
5. The electronic device of claim 1 , wherein the instructions are further configured to cause the processor to:
obtain the planar mesh by moving vertices corresponding to a same scan time point in the default mesh using a transformation matrix that is a rigid body transformation corresponding to the motion of the body.
6. The electronic device of claim 1 , wherein the instructions are further configured to cause the processor to:
obtain the planar mesh by not-moving vertices in the default mesh that correspond to a reference time point and by moving other vertices of the default mesh based on the motion of the body.
7. The electronic device of claim 1 , wherein the instructions are further configured to cause the processor to:
generate the scene view image from the image data based on a mapping relationship between image pixels of the image data and vertices of the planar mesh.
8. The electronic device of claim 7 , wherein the instructions are further configured to cause the processor to:
determine image information, corresponding to either positions of the vertices or a surrounding position on the target plane, based on pixel values of image pixels mapped to the vertices of the planar mesh in the image data; and
generate the scene view image based on the determined image information.
9. The electronic device of claim 1 , wherein image pixels mapped to at least two vertices among vertices adjacent to each other in the planar mesh are positioned along the same scan line.
10. The electronic device of claim 1 , wherein vertices included in the planar mesh are mapped to image pixels belonging to a portion of a field of view of the camera sensor overlapping the target plane.
11. The electronic device of claim 1 , wherein image pixels corresponding to a same scan line are spaced apart from each other at regular intervals along a line within the image data.
12. The electronic device of claim 1 , wherein vertices of the default mesh are disposed at positions back-projected onto the target plane along directions from a sensing element corresponding to an image pixel mapped to a corresponding vertex toward an origin of the camera sensor based on a position of the camera sensor with respect to the body.
13. The electronic device of claim 1 ,
wherein the body is a vehicle and the electronic device is mounted on, affixed to, or incorporated in the vehicle, and
wherein the camera sensor is positioned at a front surface, rear surface, or side surface of the vehicle.
14. The electronic device of claim 1 , wherein the instructions are further configured to cause the processor to:
identify a road marking from the generated scene view image.
15. The electronic device of claim 1 , wherein the instructions are further configured to cause the processor to:
generate the scene view image having a view, including at least a partial region included in a viewing angle of the camera sensor, in the target plane, from a view point of the camera sensor.
16. The electronic device of claim 1 , wherein the instructions are further configured to cause the processor to:
generate, as the scene view image, a top-view image of a region, including at least a portion of the planar mesh on the target plane, the top-view image corresponding to a view looking directly down on the target plane; and
output the top-view image through a display.
17. A processor-implemented method, the method comprising:
receiving an image, the image having been generated by a rolling shutter camera sensor that sequentially scans lines in a camera incorporated in a moving apparatus;
obtaining a planar mesh by using a default mesh comprising vertices corresponding to positions where image pixels along a scan line of the camera sensor are back-projected onto a target plane and by using a motion of the moving apparatus, the target plane being located and included in a field of view (FOV) of the camera; and
generating a scene view image from the image data based on the obtained planar mesh.
18. A method comprising:
obtaining an image captured by a rolling shutter camera that is moving through physical space, wherein lines of the image are captured at respective line capture times while the rolling shutter camera is moving;
forming a planar mesh by, for each group of vertices of an initial mesh, translating the vertices of groups in the initial mesh based on motion of the rolling shutter camera at the respective line capture times of the groups, wherein the initial mesh is obtained by back-projecting image pixels along a scan line of the rolling shutter camera onto a target plane, the target plane being located and included in a field of view (FOV) of the rolling shutter camera, and the translation of the vertices in a group depends on the motion of the rolling shutter camera at the group's line capture time; and
generating an image based on the planar mesh.
19. The method of claim 18 , wherein the vertices are arranged by back-projecting based on camera parameters of the rolling shutter camera.
20. The method of claim 18 , wherein the vertices of the initial mesh are co-planar with respect to each other, and wherein the vertices of the planar mesh are co-planar with respect to each other.