Granted Patent B2

US 12,567,201 · App. 18/412,520 · Granted Mar 3, 2026

Inventor: Vadym Suprun (Edmonton, CA)

Assignee: LEIA Inc.

G06T15/205G06T7/70G06T7/90G06T15/06G06T2207/10016G06T2207/10024G06T2207/10028G06T2207/20212G06T2210/21

Abstract

Claims (91)

1 . A computer-implemented method of multiview image generation and display, the method comprising:

receiving, with a computing device, a color image of a scene and a depth map of the scene;

synthesizing, with the computing device, view images of a multiview image of the scene from the color image and the depth map, the view images representing the scene from different view directions, each view image including pixel locations and respective color values at the pixel locations; and

rendering the view images of the multiview image on a multiview display of the computing device,

wherein synthesizing a view image of the view images of the multiview image comprises, for a pixel location of the pixel locations in the view image:

casting a ray from the pixel location toward the scene in a direction corresponding to the view direction of the view image;

determining a ray intersection location at which the ray intersects a virtual surface specified by the depth map; and

setting a color value of the respective color values of the view image at the pixel location to correspond to a color of the color image at the ray intersection location; and

wherein determining the ray intersection location comprises:

determining sequential provisional locations along the ray between the pixel location and a specified plane such that the virtual surface lies between the pixel location and the specified plane;

identifying a provisional location of the sequential provisional locations along the ray comprising:

determining that the identified provisional location lies between the pixel location and the specified plane; and

advancing the identified provisional location to a next provisional location along the ray;

determining that the virtual surface lies between the pixel location and the identified provisional location; and

setting the ray intersection location to correspond to a location between the identified provisional location and an adjacent and previously identified provisional location, inclusive.

2 . The computer-implemented method of claim 1 , wherein determining the ray intersection location further comprises:

determining sequential second provisional locations along the ray between the identified provisional location and the adjacent and previously identified provisional location;

identifying a second provisional location of the sequential second provisional locations along the ray comprising:

determining that the identified second provisional location lies between the pixel location and the specified plane; and

advancing the identified second provisional location to a next second provisional location along the ray;

determining that the virtual surface lies between the pixel location and the identified second provisional location; and

setting the ray intersection location to correspond to a location between the identified second provisional location and an adjacent and previously-identified second provisional location, inclusive.

3 . The computer-implemented method of claim 1 , wherein the provisional locations are equally spaced along the ray.

4 . The computer-implemented method of claim 3 , wherein:

the view image defines a horizontal direction being parallel to a top and bottom edges of the view image, a vertical direction being in a plane of the view image and orthogonal to the horizontal direction, and a depth that is orthogonal to the horizontal and vertical directions; and

the provisional locations are spaced such that a horizontal component of a spacing between adjacent provisional locations corresponds to a specified value.

5 . The computer-implemented method of claim 4 , wherein the specified value corresponds to a horizontal spacing between adjacent pixels in the view image.

6 . The computer-implemented method of claim 1 , wherein synthesizing a view image of the view images of the multiview image comprises, for the pixel location in the view image:

casting the ray from the pixel location toward the scene in the direction corresponding to the view direction of the view image;

determining that the ray does not intersect the virtual surface specified by the depth map;

retrieving color information from at least one time-adjacent video frame of the view images of the multiview image; and

setting the color value of the respective color values of the view image at the pixel location to correspond to the retrieved color information.

7 . The computer-implemented method of claim 6 , wherein determining that the ray does not intersect the virtual surface specified by the depth map comprises determining that the ray has propagated away from the pixel location by a distance that exceeds a threshold distance.

8 . The computer-implemented method of claim 6 , wherein the view images correspond to sequential images of a video signal; and the color information is retrieved from the pixel location of the at least one time-adjacent video frame of the video signal.

9 . The computer-implemented method of claim 1 , wherein the different view directions lie in a horizontal plane that is parallel to top and bottom edges of the view images.

10 . A system configured to perform multiview image generation and display, the system comprising:

a multiview display;

a central processing unit; and

a memory that stores a plurality of instructions, which, when executed, cause the central processing unit to perform operations, the operations comprising:

receiving a color image of a scene and a depth map of the scene;

synthesizing view images of a multiview image of the scene from the color image and the depth map, the view images representing the scene from different view directions, each view image including pixel locations and respective color values at the pixel locations; and

rendering the view images of the multiview image on the multiview display,

wherein synthesizing a view image of the view images of the multiview image comprises, for a pixel location of the pixel locations in the view image:

casting a ray from the pixel location toward the scene in a direction corresponding to the view direction of the view image;

determining a ray intersection location at which the ray intersects a virtual surface specified by the depth map; and

setting a color value of the respective color values of the view image at the pixel location to correspond to a color of the color image at the ray intersection location; and

wherein determining the ray intersection location comprises:

determining sequential provisional locations along the ray between the pixel location and a specified plane such that the virtual surface lies between the pixel location and the specified plane;

identifying a provisional location of the sequential provisional locations along the ray comprising:

determining that the identified provisional location lies between the pixel location and the specified plane; and

advancing the identified provisional location to a next provisional location along the ray;

determining that the virtual surface lies between the pixel location and the identified provisional location; and

setting the ray intersection location to correspond to a location between the identified provisional location and an adjacent and previously identified provisional location, inclusive.

11 . The system of claim 10 , wherein determining the ray intersection location further comprises:

determining sequential second provisional locations along the ray between the identified provisional location and the adjacent and previously identified provisional location;

identifying a second provisional location of the sequential second provisional locations along the ray comprising:

determining that the identified second provisional location lies between the pixel location and the specified plane; and

advancing the identified second provisional location to a next second provisional location along the ray;

determining that the virtual surface lies between the pixel location and the identified second provisional location; and

setting the ray intersection location to correspond to a location between the identified second provisional location and an adjacent and previously-identified second provisional location, inclusive.

12 . The system of claim 10 , wherein the provisional locations are equally spaced along the ray.

13 . The system of claim 12 , wherein:

the view image defines a horizontal direction being parallel to a top and bottom edges of the view image, a vertical direction being in a plane of the view image and orthogonal to the horizontal direction, and a depth that is orthogonal to the horizontal and vertical directions; and

the provisional locations are spaced such that a horizontal component of a spacing between adjacent provisional locations corresponds to a horizontal spacing between adjacent pixels in the view image.

14 . The system of claim 10 , wherein the different view directions lie in a horizontal plane that is parallel to top and bottom edges of the view images.

15 . A non-transitory, computer-readable storage medium storing executable instructions that, when executed by a processor of a computer system, perform operations of multiview image generation and display, the operations comprising:

receiving a color image of a scene and a depth map of the scene;

synthesizing view images of a multiview image of the scene from the color image and the depth map, the view images representing the scene from different view directions, each view image including pixel locations and respective color values at the pixel locations; and

rendering the view images of the multiview image on a multiview display,

wherein synthesizing a view image of the view images of the multiview image comprises, for a pixel location of the pixel locations in the view image:

casting a ray from the pixel location toward the scene in a direction corresponding to the view direction of the view image;

determining a ray intersection location at which the ray intersects a virtual surface specified by the depth map; and

setting a color value of the respective color values of the view image at the pixel location to correspond to a color of the color image at the ray intersection location; and

wherein determining the ray intersection location comprises:

determining sequential provisional locations along the ray between the pixel location and a specified plane such that the virtual surface lies between the pixel location and the specified plane;

identifying a provisional location of the sequential provisional locations along the ray comprising repeatedly:

determining that the identified provisional location lies between the pixel location and the specified plane; and

advancing the identified provisional location to a next provisional location along the ray;

determining that the virtual surface lies between the pixel location and the identified provisional location; and

setting the ray intersection location to correspond to a location between the identified provisional location and an adjacent and previously identified provisional location.

16 . The non-transitory, computer-readable storage medium of claim 15 , wherein determining the ray intersection location further comprises:

determining sequential second provisional locations along the ray between the identified provisional location and the adjacent and previously identified provisional location;

identifying a second provisional location of the sequential second provisional locations along the ray comprising repeatedly:

determining that the identified second provisional location lies between the pixel location and the specified plane; and

advancing the identified second provisional location to a next second provisional location along the ray;

determining that the virtual surface lies between the pixel location and the identified second provisional location; and

setting the ray intersection location to correspond to a location between the identified second provisional location and an adjacent and previously-identified second provisional location, inclusive.

17 . The non-transitory, computer-readable storage medium of claim 15 , wherein:

the provisional locations are equally spaced along the ray;

the view image defines a horizontal direction being parallel to a top and bottom edges of the view image, a vertical direction being in a plane of the view image and orthogonal to the horizontal direction, and a depth that is orthogonal to the horizontal and vertical directions; and

the provisional locations are spaced such that a horizontal component of a spacing between adjacent provisional locations corresponds to a horizontal spacing between adjacent pixels in the view image.

Assignments (2)

Continuity (3)

References Cited (54)