Granted Patent B2

US 12,483,685 · App. 18/862,537 · Granted Nov 25, 2025

Inventor: Christiaan Varekamp (Veldhoven, NL)

Assignee: Koninklijke Philips N.V.

H04N13/282G06T3/40G06T7/579G06T7/80H04N13/111H04N13/257G06T2207/10024G06T2207/10028

Abstract

Claims (81)

1 . A method comprising:

obtaining a depth map from each of at least one depth sensors;

obtaining a plurality of images from at least two color sensors;

generating at least two up to scale camera poses for the at least two color sensors;

generating an up to scale point cloud using the plurality of images;

estimating a scaling of the at least two up to scale camera poses and up to scale point cloud using the at least one depth maps and

obtaining a plurality of proposed scales; for each proposed scale:

calculating at least two proposed camera poses and a proposed point cloud by applying the proposed scale to the at least two up to scale camera poses and up to scale point cloud respectively;

comparing the proposed point cloud to the at least one depth maps;

determining at least one error metrics based on the comparison; and

estimating the scaling based on the proposed scale with the lowest at least one error metrics.

2 . The method of claim 1 ,

wherein generating the at least two up to scale camera poses and the up to scale point cloud comprises inputting the plurality of images into a structure from a motion algorithm,

wherein the structure from the motion algorithm is arranged to output the at least two up to scale camera poses for the at least one color sensors.

3 . The method of claim 1 ,

wherein comparing the proposed point cloud to the at least one depth maps comprises:

projecting the proposed point cloud to the views of the at least one depth sensors using the proposed camera poses; and

evaluating a plurality of depth-based errors between depth values in the at least one depth maps and corresponding depth values resulting from projecting the proposed point cloud, wherein determining one of the error metrics is based on the plurality of depth-based errors.

4 . The method of claim 1 , wherein comparing the proposed point cloud comprises:

projecting the proposed point cloud to the views of the at least two color sensors using the at least two proposed camera poses so as to generate projected points;

identifying depth map values of the at least one depth maps corresponding to feature points of the proposed point cloud;

projecting the depth map values to the views of the at least two color sensors using the at least two proposed camera poses; and

evaluating a plurality of pixel-based errors, in an imaging plane of each of the at least two color sensors between the pixel coordinates resulting from projecting the depth map values and the pixel coordinates resulting from projecting the proposed point cloud,

wherein determining one of the error metrics is based on the plurality of pixel-based errors.

5 . The method of claim 1 ,

wherein estimating the scaling of the at least two up to scale camera poses and up to scale point cloud comprises, for each proposed scale:

determining whether one or at least one projected points in the projected point cloud are occluded in the view of any one of the depths sensors and/or the at least two color sensors; and

based on a particular projected point being occluded, weighting the error corresponding to the particular projected point lower in the at least one error metrics.

6 . The method of claim 3 ,

wherein estimating the scaling of the at least two up to scale camera poses and up to scale point cloud further comprises, for each proposed scale:

determining whether one or at least one projected points in the projected point cloud are occluded in the view of any one of the depths sensors and/or the at least two color sensors; and

based on a particular projected point occluded, weighting the error corresponding to the particular projected point lower in the at least one error metrics,

wherein determining whether at least one projected points in the projected point cloud are occluded comprises determining whether the depth-based error between the depth of at least one of the projected points and the depth of the depth map exceeds a depth-based occlusion threshold.

7 . The method of claim 4 ,

wherein estimating the scaling of the at least two up to scale camera poses and up to scale point cloud further comprises, for each proposed scale:

determining whether one or at least one projected points in the projected point cloud are occluded in the view of any one of the depths sensors and/or the at least two color sensors; and

based on a particular projected point being occluded, weighting the error corresponding to the particular projected point lower in the at least one error metrics, and

wherein determining whether at least one projected points in the projected point cloud are occluded comprises determining whether the pixel-based error between the pixel coordinates of the projected points and the pixel coordinates of the depth map points exceeds a pixel-based occlusion threshold.

8 . The method of claim 5 ,

wherein determining whether at least one projected points in the projected point cloud are occluded comprises:

identifying projected points within a local neighborhood of at least one central projected points; and

determining whether the depth-based error and/or the pixel-based error exceed a neighborhood threshold, wherein the neighborhood threshold is optionally dependent on the depth of the identified projected points in the local neighborhood for each local neighborhood.

9 . The method of claim 5 ,

wherein determining whether at least one projected points in the projected point cloud are occluded comprises:

identifying, for each projected point, a pixel color for each view of the at least one depth sensors and/or the at least two color sensors based on the plurality of images for each projected point; and

comparing the pixel colors from each view for each projected point.

10 . The method of claim 1 , further comprising:

generating a second point cloud using the at least one depth maps;

for each proposed scale:

projecting the second point cloud to the views of the at least one depth sensors using the at least two proposed camera poses; and

evaluating a second plurality of depth-based errors between depth values in the at least one depth maps and the corresponding depth values resulting from the projection of the second point cloud,

wherein determining one of the error metrics is based on the second plurality of depth-based errors.

11 . A computer program stored on a non-transitory medium, wherein the computer program when executed on a processor performs the method as claimed in claim 1 .

12 . A system, the system comprising:

a hybrid camera array comprising at least two color sensors and at least one depth sensors; and

a processor circuit and a memory circuit, wherein the memory is arranged to store instructions for the processor circuit,

wherein the processor circuit is arranged to obtain a depth map from each of the at least one depth sensors,

wherein the processor circuit is arranged to obtain a plurality of images from the at least two color sensors,

wherein the processor circuit is arranged to generate at least two up to scale camera poses for the at least two color sensors,

wherein the processor circuit is arranged to an up to scale point cloud using the plurality of images,

wherein the processor circuit is arranged to estimate a scaling of the at least two up to scale camera poses,

wherein the processor circuit is arranged to estimate up to scale point cloud using the at least one depth maps,

wherein the estimating comprises:

wherein the processor circuit is arranged to obtain a plurality of proposed scales,

wherein the processor circuit is arranged to calculate at least two proposed camera poses and a proposed point cloud by applying the proposed scale to the at least two up to scale camera poses and up to scale point cloud respectively for each proposes scale,

wherein the processor circuit is arranged to compare the proposed point cloud to the at least one depth maps for each proposed scale,

determining at least one error metrics based on the comparison for each proposed scale,

estimating the scaling based on the proposed scale with the lowest at least one error metrics for each proposed scale.

13 . The system of claim 12 ,

wherein the processor circuit is arranged to generate the at least two up to scale camera poses,

wherein the processor circuit is arranged to generate the up to scale point cloud by inputting the plurality of images into a structure from a motion algorithm,

wherein the structure from the motion algorithm is arranged to output the at least two up to scale camera poses for the at least two color sensors.

14 . The system of claim 12 ,

wherein the processor circuit is arranged to generate the at least two up to scale camera poses,

wherein the processor circuit is arranged to generate the up to scale point cloud by inputting the plurality of images into a structure from a motion algorithm,

wherein the structure from the motion algorithm is arranged to output the at least two up to scale camera poses for the at least two color sensors,

wherein the structure from the motion algorithm is arranged to output the up to scale point cloud.

15 . The method of claim 1 ,

wherein generating the at least two up to scale camera poses and the up to scale point cloud comprises inputting the plurality of images into a structure from a motion algorithm,

wherein the structure from the motion algorithm is arranged to output the at least two up to scale camera poses for the at least one color sensors,

wherein the structure from the motion algorithm is arranged to output the up to scale point cloud.

Assignments (1)

Priority Claims (1)

Continuity (1)

References Cited (20)