Granted Patent B1

US 12,243,250 · App. 18/519,154 · Granted Mar 4, 2025

Inventor: Henry Harlyn Baker (Los Altos, CA)

G06T7/50G06T7/13G06T7/593G06T7/70G06T2207/10012G06T2207/10024G06T2207/30201

Abstract

Claims (54)

1. An apparatus to facilitate synthesis of a gaze-aligned view of a face of a human from the perspective of a viewer, the apparatus comprising:

three or more cameras positionally arranged in common plane, each of the three or more cameras in a positional relationship so as to capture a respective perspective view of the face, a first subset of the three or more cameras forming a first linear camera array and a second subset of the three or more cameras forming a second linear camera array, the first linear camera array and the second linear camera array corresponding to independent dimensions of the common plane;

a frame or chassis that mechanically mounts the three or more cameras in the positional relationship;

wherein eyes of the face define a horizontal axis, which intersects the eyes, and a vertical axis, which bisects the eyes and which passes through the horizontal axis;

wherein the three or more cameras are positioned such that the respective views of the face correspond to perspectives taken of the face from each side of the vertical axis and perspective views of the face taken from each side of the horizontal axis; and

at least one electrical interface to couple electrical outputs of the three or more cameras, corresponding to images of the face, with at least one processor, the at least one processor to perform the synthesis of the gaze-aligned view of the face using the images taken of the face.

2. The apparatus of claim 1 wherein the frame or chassis is structured such that neither it nor any camera of the three or more cameras blocks a ray which extends parallel to the gaze-aligned view of the face and passes through an intersection of the horizontal axis and the vertical axis and wherein the viewer is an individual viewing an electronic display device.

3. The apparatus of claim 1 wherein: the apparatus comprises an electronic display device; the frame or chassis is mechanically coupled to the electronic display device; the at least one processor is further configured to display an image on the electronic display device; and the frame or chassis mounts each camera of the three or more cameras outside of a region imaged by the electronic display device.

4. The apparatus of claim 3 , embodied as a live video call device, wherein the apparatus is configured to transmit, to a recipient device, an image of the gaze-aligned view, and wherein the apparatus is configured to display, on the electronic display device, an image received from the recipient device.

5. The apparatus of claim 1 wherein the apparatus further comprises the at least one processor.

6. The apparatus of claim 5 wherein the at least one processor is configured to:

receive digital values corresponding to a first image from a first camera of the three or more cameras, a second image from a second one of the three or more cameras, and a third image from a third one of the three or more cameras;

create a first epipolar-plane image from the digital values;

process the first epipolar-plane image, by identifying a plurality of edges in the first epipolar-plane image and by fitting a line to the plurality of edges;

synthesize a three-dimensional model of the face from the fitted line and from image data from at least one of the three or more cameras; and

create the gaze-aligned view of the face using the three-dimensional model.

7. The apparatus of claim 6 wherein one of the first camera, the second camera and the third camera is a reference camera, wherein a corresponding one of the first image, the second image and the third image is a reference image, and wherein the at least one processor is configured to synthesize the gaze-aligned view using the fitted line and the reference image.

8. The apparatus of claim 7 wherein the reference camera is a camera which is in each of the first linear camera array and the second linear camera array.

9. The apparatus of claim 1 wherein the apparatus further comprises electronic storage, wherein the at least one processor is to store an image of the gaze-aligned view of the face, in digital electronic form, in the electronic storage, and wherein the apparatus further comprises circuitry to transmit the digital, electronic form of the gaze-aligned view of the face to remote electronic circuitry.

10. The apparatus of claim 1 wherein the three or more cameras are mechanically mounted, by the frame or chassis, such that the positional relationship corresponds to form a polygon.

11. An apparatus to synthesize a gaze-aligned view of a face of a human from the perspective of a viewer, the apparatus comprising:

a camera assembly, the camera assembly having

three or more cameras positionally arranged in common plane, each of the three or more cameras in a positional relationship so as to capture a respective perspective view of the face, a first subset of the three or more cameras forming a first linear camera array and a second subset of the three or more cameras forming a second linear camera array, the first linear camera array and the second linear camera array corresponding to independent dimensions of the common plane,

a frame or chassis that mechanically mounts the three or more cameras in the positional relationship, and

at least one electrical interface;

wherein

eyes of the face define a horizontal axis, which intersects the eyes, and a vertical axis, which bisects the eyes and which passes through the horizontal axis, and

the three or more cameras are positioned such that the respective views of the face correspond to perspectives taken of the face from each side of the vertical axis and perspective views of the face taken from each side of the horizontal axis;

electronic storage; and

at least one processor configured to receive electrical outputs of the three or more cameras, corresponding to images of the face, from the electrical interface, to synthesize the gaze-aligned view of the face using the images taken of the face, and to store an image of the gaze-aligned view of the face in the electronic storage.

12. The apparatus of claim 11 wherein the frame or chassis is structured such that neither it nor any camera of the three or more cameras blocks a ray which extends parallel to the gaze-aligned view of the face and passes through an intersection of the horizontal axis and the vertical axis, and wherein the viewer is an individual viewing the image.

13. The apparatus of claim 11 wherein: the apparatus comprises an electronic display device; the frame or chassis is mechanically coupled to the electronic display device; the at least one processor is further configured to display an image on the electronic display device; and the frame or chassis mounts each camera of the three or more cameras outside of a region imaged by the electronic display device.

14. The apparatus of claim 13 , embodied as a live video call device, wherein the apparatus is configured to transmit, to a recipient device, an image of the gaze-aligned view, and wherein the apparatus is configured to display, on the electronic display device, an image received from the recipient device.

15. The apparatus of claim 14 wherein the three or more cameras are mechanically mounted, by the frame or chassis, such that the positional relationship corresponds to form a polygon, the electronic display device positioned within the polygon.

16. The apparatus of claim 11 wherein the at least one processor is configured to:

receive digital values corresponding to a first image from a first camera of the three or more cameras, a second image from a second one of the three or more cameras, and a third image from a third one of the three or more cameras;

create a first epipolar-plane image from the digital values;

process the first epipolar-plane image, by identifying a plurality of edges in the first epipolar-plane image and by fitting a line to the plurality of edges;

synthesize a three-dimensional model of the face from the fitted line and from image data from at least one of the three or more cameras; and

create the gaze-aligned view of the face using the three-dimensional model.

17. The apparatus of claim 16 wherein one of the first camera, the second camera and the third camera is a reference camera, wherein a corresponding one of the first image, the second image and the third image is a reference image, and wherein the at least one processor is configured to synthesize the gaze-aligned view using the fitted line and the reference image.

18. The apparatus of claim 17 wherein the reference camera is a camera which is in each of the first linear camera array and the second linear camera array.

19. A video conferencing apparatus to synthesize a gaze-aligned view of a face of a first human from the perspective of a viewer, the video conferencing apparatus comprising:

an electronic display device to display video received from a recipient device associated with the viewer;

a camera assembly, the camera assembly having

three or more cameras positionally arranged in common plane, each of the three or more cameras in a positional relationship so as to capture a respective perspective view of the face, a first subset of the three or more cameras forming a first linear camera array and a second subset of the three or more cameras forming a second linear camera array, the first linear camera array and the second linear camera array corresponding to independent dimensions of the common plane,

a frame or chassis that mechanically mounts the three or more cameras in the positional relationship, and

at least one electrical interface;

wherein

eyes of the face define a horizontal axis, which intersects the eyes, and a vertical axis, which bisects the eyes,

the frame is structured such that neither it nor any camera of the three or more cameras blocks a ray which extends parallel to the gaze-aligned view of the face, passes through an intersection of the horizontal axis and the vertical axis, and intersects the electronic display device, and

the three or more cameras are positioned such that the respective views of the face correspond to perspectives taken of the face from each side of the vertical axis and perspective views of the face taken from each side of the horizontal axis;

electronic storage; and

at least one processor configured to receive electrical outputs of the three or more cameras, corresponding to images of the face, from the electrical interface, to synthesize the gaze-aligned view of the face using the images taken of the face, to store an image of the gaze-aligned view of the face in the electronic storage, and to transmit the image from the electronic storage to the recipient device.

Continuity (5)

References Cited (109)