Virtual 3D methods, systems and software
Methods, systems and computer program products (“software”) enable a virtual three-dimensional visual experience (referred to herein as “V3D”) in videoconferencing and other applications, and capturing, processing and displaying of images and image streams.
1 . A video communication method that enables a first user to view a second user with direct virtual eye contact with the second user, the method comprising:
capturing images of the second user, the capturing comprising utilizing at least one camera having a view of the second user's face;
executing a feature correspondence function by detecting common features between corresponding images captured by the at least one camera and measuring a relative distance in image space between the common features, to generate disparity values;
generating a data representation, representative of the captured images and the corresponding disparity values;
reconstructing a synthetic view of the second user, based on the representation; and
displaying the synthetic view to the first user on a display screen used by the first user;
the capturing, detecting, generating, reconstructing and displaying being executed such that the first user can have direct virtual eye contact with the second user through the first user's display screen, by the reconstructing and displaying of the synthetic view of the second user in which the second user appears to be gazing directly at the first user, even if the at least one camera does not have a direct eye contact gaze vector to the second user.
2 . The method of claim 1 wherein the capturing comprises utilizing at least two cameras, each having a view of the second user's face; and
wherein executing a feature correspondence function comprises detecting common features between corresponding images captured by the respective cameras.
3 . The method of claim 1 wherein:
the capturing comprises utilizing at least one camera having a view of the second user's face and which is an infra-red time-of-flight camera that directly provides depth information; and
the data representation is representative of the captured images and corresponding depth information.
4 . The method of claim 1 wherein:
the capturing comprises utilizing a single camera having a view of the second user's face; and
executing a feature correspondence function comprises detecting common features between images captured by the single camera over time and measuring a relative distance in image space between the common features, to generate the disparity values.
5 . The method of claim 2 wherein:
the captured images of the second user comprise visual information of the scene surrounding the second user; and
the capturing, detecting, generating, reconstructing and displaying are executed such that:
(a) the first user is provided the visual impression of looking through his display screen as a physical window to the second user and the visual scene surrounding the second user, and
(b) the first user is provided an immersive visual experience of the second user and the scene surrounding the second user.
6 . The method of claim 5 further comprising:
executing image rectification to compensate for optical distortion and relative misalignment of each of the at least one camera.
7 . The method of claim 6 wherein executing image rectification comprises applying a 2D image space transform.
8 . The method of claim 7 wherein applying a 2D image space transform comprises utilizing a GPGPU processor running a shader program.
9 . The method of claim 5 wherein:
the at least two cameras for capturing images of the second user are located at or near the periphery or edges of a display device used by the second user, the display device used by the second user having a display screen viewable by the second user and having a geometric center, and
the synthetic view of the second user corresponds to a selected virtual camera location, the selected virtual camera location corresponding to a point or proximate to the geometric center.
10 . The method of claim 5 wherein the at least two cameras for capturing images of the second user are located at a selected position outside the periphery or edges of a display device used by the second user.
11 . The method of claim 5 wherein the at least two cameras for capturing images of the second user are located in selected positions and positioned with selected orientations around the second user.
12 . The method of claim 5 wherein the at least two cameras for capturing images of the second user are located in selected positions and positioned with selected orientations around the second user.
13 . The method of claim 5 further comprising:
estimating a location of the first user's head or eyes, thereby generating tracking information; and
wherein the reconstructing of a synthetic view of the second user comprises reconstructing the synthetic view based on the generated data representation and the generated tracking information.
14 . The method of claim 5 wherein:
camera shake effects are inherently eliminated, in that the capturing, detecting, generating, reconstructing and displaying are executed such that the first user has a virtual direct view through his display screen to the second user and a visual scene surrounding the second user; and
scale and perspective of the image of the second user and objects in the visual scene surrounding the second user are accurately represented to the first user regardless of a user view distance and angle.
15 . The method of claim 5 wherein the method is adapted for implementation on a mobile telephone device, and the at least two cameras for capturing images of the second user are located at or near the periphery or edges of a mobile telephone device used by the second user.
16 . The method of claim 5 wherein the method is adapted for implementation on a laptop or desktop computer, and the at least two cameras for capturing images of the second user are located at or near the periphery or edges of a display device of a laptop or desktop computer by the second user.
17 . The method of claim 5 wherein the method is adapted for implementation on a computing or telecommunications devices comprising any of table computing devices computer-driven television displays or computer-driven image projection devices, and wherein the at least two cameras for acquiring images of the second user are located at or near the periphery or edges of a computing or telecommunications device used by the second user.
18 . The method of claim 5 wherein the capturing comprises utilizing exposure cycling, the exposure cycling comprising:
dynamically adjusting the exposure of the at least two cameras on a frame-by-frame basis to improve disparity estimation in regions outside the exposed region viewed by the user;
wherein a series of exposures are taken, including exposures lighter than and exposures darker than a visibility-optimal exposure, disparity values are calculated for each exposure, and the disparity values are integrated into an overall disparity solution over time, so as to improve disparity estimation.
19 . The method of claim 5 wherein the capturing comprises utilizing exposure cycling, the exposure cycling comprising:
dynamically adjusting an exposure setting of the at least two cameras on a frame-by-frame basis to provide a disparity estimation in regions outside the exposed region viewed by the user;
wherein a series of exposures are taken, including exposures lighter than and exposures darker than a visibility-optimal exposure, disparity values are calculated for each exposure, and the disparity values are integrated into a disparity histogram, the disparity histogram being converged over time, so as to improve the disparity estimation.
20 . The method of claim 19 further comprising analyzing a quality of the disparity estimation on a respective dark, mid-range and light pixels to generate variance information used to control the exposure settings of the at least two cameras, thereby to form a closed loop between the quality of the disparity estimate and the exposure settings.
21 . The method of claim 20 further comprising analyzing variance of the disparity histograms on respective dark, mid-range and light pixels to generate variance information used to control the exposure settings of the at least two cameras, thereby to form a closed loop between the quality of the disparity estimate and the exposure settings.
22 . The method of claim 5 wherein the feature correspondence function comprises filling unknowns in a correspondence information set with historical data obtained from previously captured images.
23 . The method of claim 22 wherein the filling of unknowns comprises:
if a given image feature is detected in an image captured by one of the cameras, and no corresponding image feature is found in a corresponding image captured by another of the cameras, then utilizing data for a pixel corresponding to the given image feature, from a corresponding, previously captured image.
24 . A program product for use with a digital processing system, for enabling a first user to view a second user with direct virtual eye contact with the second user, the digital processing system comprising at least one camera having a view of the second user's face, a display screen for use by the first user, and a digital processing resource comprising at least one digital processor, the program product comprising digital processor-executable program instructions stored on a non-transitory digital processor-readable medium, which when executed in the digital processing resource cause the digital processing resource to:
capture images of the second user, utilizing the at least one camera;
execute a feature correspondence function by detecting common features between corresponding images captured by the at least one camera and measuring a relative distance in image space between the common features, to generate disparity values;
generate a data representation, representative of the captured images and the corresponding disparity values;
reconstruct a synthetic view of the second user, based on the representation; and
display the synthetic view to the first user on the display screen for use by the first user;
the capturing, detecting, generating, reconstructing and displaying being executed such that the first user can have direct virtual eye contact with the second user through the first user's display screen, by the reconstructing and displaying of a synthetic view of the second user in which the second user appears to be gazing directly at the first user, even if no camera has a direct eye contact gaze vector to the second user.
25 . The program product of claim 24 further comprising digital processor-executable program instructions stored on a non-transitory digital processor-readable medium, which when executed in the digital processing resource cause the digital processing resource to:
estimate a location of the first user's head or eyes, thereby generating tracking information; and
wherein the reconstructing of a synthetic view of the second user, based on the representation comprises reconstructing the synthetic view based on the generated data representation and the generated tracking information.
26 . A digital processing system for enabling a first user to view a second user with direct virtual eye contact with the second user, the digital processing system comprising:
at least one camera having a view of the second user's face;
a display screen for use by the first user; and
a digital processing resource comprising at least one digital processor, the digital processing resource being operable to:
capture images of the second user, utilizing the at least one camera;
executing a feature correspondence function by detecting common features between corresponding images captured by the at least one camera and measuring a relative distance in image space between the common features, to generate disparity values;
generate a data representation, representative of the captured images and the corresponding disparity values;
reconstruct a synthetic view of the second user, based on the representation; and
display the synthetic view to the first user on the display screen for use by the first user;
the capturing, detecting, generating, reconstructing and displaying being executed such that the first user can have direct virtual eye contact with the second user through the first user's display screen, by the reconstructing and displaying of a synthetic view of the second user in which the second user appears to be gazing directly at the first user, even if no camera has a direct eye contact gaze vector to the second user.
27 . The digital processing system of claim 26 wherein the digital processing resource is operable to:
estimate a location of the first user's head or eyes, thereby generating tracking information; and
wherein the reconstructing of a synthetic view of the second user, based on the representation comprises reconstructing the synthetic view based on the generated data representation and the generated tracking information.