Granted Patent B2

US 12,263,836 · App. 18/500,174 · Granted Apr 1, 2025

Inventors: Leaf Alden Jiang (Concord, MA); Carsten Louis Boers (Cambridge, MA)

Assignee: NODAR Inc.

B60W30/09G01S17/89G06T7/50G06T2207/10024G06T2207/10028

Abstract

Claims (46)

1. A system comprising:

a first sensor system configured to:

receive first image data of a scene captured by a pair of camera sensors,

calculate first disparity values and first confidence values from the first image data, and

output a first disparity map comprising the first disparity values, and output a first confidence map comprising the first confidence values; and

a machine control system configured to output a first control signal to a control unit of the machine to control the machine based on the first disparity map and the first confidence map, wherein controlling the machine based on the first disparity map and the first confidence map comprises determining whether to use the first disparity values or discard the first disparity values based on an assessment of the corresponding first confidence values.

2. The system of claim 1 , wherein the machine control system is further configured to output, to a display device of the machine, a video stream based on the first disparity map and the first confidence map.

3. The system of claim 1 , wherein:

the first image data comprises a plurality of pixels,

the first disparity map comprises a disparity value for each of the pixels, and

the first confidence map comprises a confidence value for each of the pixels.

4. The system of claim 1 , further comprising:

a second sensor system configured to receive second image data of at least a portion of the scene and to output a second confidence map based on the second image data,

wherein the machine control system is further configured to output a video stream as a sequence of super-frames, with each super-frame comprising information based on: the first disparity map, the first confidence map, and the second confidence map.

5. The system of claim 1 , wherein the machine control system is configured to output a plurality of control signals to a plurality of control units of the machine based on the first disparity map and the first confidence map.

6. The system of claim 4 , wherein:

the video stream comprises:

at least one super-frame comprising the first disparity map and the first confidence map, and

at least one super-frame comprising the first disparity map and the second confidence map.

7. The system of claim 4 , wherein the video stream comprises at least one super-frame comprising a portion of the first confidence map and a portion of the second confidence map.

8. The system of claim 4 , wherein:

the first image data comprises stereo-vision data, and

the second image data comprises lidar data.

9. The system of claim 1 , wherein the first sensor system comprises a confidence processor configured:

to generate a plurality of camera-based confidence maps based on the first image data,

to aggregate data from the plurality of camera-based confidence maps, and

to output the first confidence map based on the data aggregated from the plurality of camera-based confidence maps.

10. The system of claim 2 , wherein:

the video stream comprises a plurality of super-frames, and

each super-frame of the video stream comprises:

a first region corresponding to a two-dimensional (2D) image of the scene,

a second region corresponding to a depth map of the scene, and

a third region corresponding to a certainty map of the scene.

11. The system of claim 10 , wherein pixels of the 2D image of the scene, pixels of the depth map of the scene, and pixels of the certainty map of the scene are temporally and spatially matched.

12. The system of claim 2 , wherein the video stream comprises two-dimensional (2D) color images, with each 2D color image comprising a plurality of pixels, and with an alpha-channel transparency of each pixel being proportional to a confidence value for the pixel.

13. A non-transitory computer-readable storage medium storing code that, when executed by a computer processor, causes the computer processor to perform a method, wherein the method comprises:

the computer processor receiving first image data of a scene captured by a pair of camera sensors,

the computer processor calculating first disparity values and first confidence values from the first image data,

the computer processor generating a first disparity map comprising the first disparity values, and generating a first confidence map comprising the first confidence values, and

the computer processor outputting a first control signal to a control unit of a machine to control the machine based on the first disparity map and the first confidence map, wherein controlling the machine based on the first disparity map and the first confidence map comprises determining whether to use the first disparity values or discard the first disparity values based on an assessment of the corresponding first confidence values.

14. The computer-readable storage medium of claim 13 , wherein the method further comprises:

the computer processor obtaining a second confidence map corresponding to second image data of at least a portion of the scene, and

the computer processor outputting a video stream as a sequence of super-frames, with each super-frame comprising information based on: the first disparity map, the first confidence map, and the second confidence map.

15. The computer-readable storage medium of claim 13 , wherein the method further comprises:

the computer processor outputting a plurality of control signals to a plurality of control units of the machine based on the first disparity map and the first confidence map.

16. The system of claim 1 , wherein the assessment of the corresponding first confidence values comprises determining whether the corresponding first confidence value is within an acceptable confidence threshold.

Assignments (1)

Continuity (6)

References Cited (114)