Granted Patent B2

US 12,568,193 · App. 17/657,724 · Granted Mar 3, 2026

Inventors: Daniele Perrone (Noisy-le-Grand, FR); Jacques Manderschied (Paris, FR); Stephane Valente (Paris, FR)

Assignee: PROPHESEE

H04N13/128G06T7/32H04N13/25G06T2207/10012

Abstract

Claims (98)

1 . A method for pixel-by-pixel registration of an event-based camera to a frame-based camera, the method comprising:

acquiring an image frame from the frame-based camera, using an exposure period;

rectifying the image frame to generate a rectified image;

acquiring a pixel event stream generated during the exposure period;

generating a first point matrix from pixel events of the pixel event stream, wherein each point of the first point matrix integrates the variation contributions of all events of the corresponding pixel during the exposure period;

rectifying the first point matrix to generate a rectified first point matrix;

computing a comparison function by comparing the rectified image and the rectified first point matrix modified by a disparity map;

computing a smoothness scoring function applied to the comparison function while varying the disparity map;

selecting the disparity map that optimizes the smoothness scoring function; and

performing pixel-by-pixel registration by applying the selected disparity map to establish correspondence between pixels of the rectified image and the rectified first point matrix.

2 . The method of claim 1 , further comprising the step of generating a deblurred image at a selected time within the exposure period, wherein:

a deblurring correction matrix is computed as a function of the selected disparity map and the selected time;

the deblurring correction matrix is unrectified to project it onto a coordinate plane of the frame-based camera; and

the deblurred image is computed as an element-by-element ratio of the rectified first point matrix and the deblurring correction matrix.

3 . The method of claim 1 , wherein the smoothness scoring function is one of the following functions:

a norm of 2D gradients of the comparison function,

a variance of local patches of the comparison function, and

a cross-correlation function applied to the rectified image and the rectified first point matrix.

4 . A method for pixel-by-pixel registration of an event-based camera to a frame-based camera, the method comprising:

acquiring an image frame from the frame-based camera, using an exposure period;

rectifying the image frame to generate a rectified image;

acquiring a pixel event stream generated during the exposure period;

generating a first point matrix from pixel events of the pixel event stream, wherein each point of the first point matrix integrates the variation contributions of all events of the corresponding pixel during the exposure period;

rectifying the first point matrix to generate a rectified first point matrix;

computing a comparison function by comparing the rectified image and the rectified first point matrix modified by a disparity map;

computing a smoothness scoring function applied to the comparison function while varying the disparity map;

selecting the disparity map that optimizes the smoothness scoring function; and

generating a deblurred image at a selected time within the exposure period by applying the selected disparity map, wherein:

the rectified first point matrix represents a deblurring correction matrix computed as a function of the selected disparity map and the selected time; and

the deblurred image is computed as an element-by-element ratio of the rectified image and the rectified first point matrix.

5 . A stereo camera device comprising:

a frame-based camera;

an event-based camera; and

a processor receiving and processing signals provided by the frame-based and event-based cameras, the processor configured to:

acquire an image frame from a frame-based camera using an exposure period;

rectify the image frame to generate a rectified image;

acquire a pixel event stream generated during the exposure period;

generate a first point matrix from pixel events of the pixel event stream, wherein each point of the first point matrix integrates the variation contributions of all events of the corresponding pixel during the exposure period;

rectify the first point matrix to generate a deblurring correction rectified first point matrix;

compute a comparison function by comparing the rectified image and the rectified first point matrix modified by a disparity map;

compute a smoothness scoring function applied to the comparison function while varying the disparity map;

select the disparity map that optimizes the smoothness scoring function; and

perform pixel-by-pixel registration by applying the selected disparity map to establish correspondence between pixels of the rectified image and the rectified first point matrix.

6 . The stereo camera device of claim 5 , wherein the processor is further configured to generate a deblurred image at a selected time within the exposure period, wherein:

a deblurring correction matrix is computed as a function of the selected disparity map and the selected time;

the deblurring correction matrix is unrectified to project it onto a coordinate plane of the frame-based camera; and

the deblurred image is computed as an element-by-element ratio of the rectified first point matrix and the deblurring correction matrix.

7 . The stereo camera device of claim 5 , wherein the smoothness scoring function is one of the following functions:

a norm of 2D gradients of the comparison function,

a variance of local patches of the comparison function, and

a cross-correlation function applied to the rectified image and the rectified first point matrix.

8 . The method of claim 1 , wherein an element-by-element comparison between the rectified image and the rectified first point matrix is an element-by-element ratio of the rectified image and the rectified first point matrix.

9 . A method for pixel-by-pixel registration of an event-based camera to a frame-based camera, the method comprising:

acquiring successive image frames from the frame-based camera over an acquisition period;

generating a first point matrix from the successive image frames, wherein each point of the first point matrix represents a variation in brightness of a corresponding pixel location of the successive image frames;

rectifying the first point matrix to generate a rectified first point matrix;

acquiring a pixel event stream generated during the acquisition period;

generating a second point matrix from pixel events of the pixel event stream, wherein each point of the second point matrix represents an aggregation of event data corresponding to a pixel location over the acquisition period;

rectifying the second point matrix to generate a rectified second point matrix;

computing a comparison function by comparing the rectified first point matrix and the rectified second point matrix modified by a disparity map;

computing a correlation scoring function applied to the comparison function while varying the disparity map; and

selecting the disparity map that optimizes the correlation scoring function; and

performing pixel-by-pixel registration by applying the selected disparity map to establish correspondence between pixel events from the event-based camera and pixels in the image frames from the frame-based camera.

10 . The method of claim 9 , wherein:

the first point matrix is generated by computing brightness ratio changes between successive image frames acquired from the frame-based camera during the acquisition period; and;

the second point matrix is generated by integrating brightness ratio changes derived from pixel events of the pixel event stream over the acquisition period.

11 . The method of claim 9 , wherein:

the first point matrix is generated by assigning a binary value for each point based on whether a brightness change at a corresponding pixel location between successive image frames exceeds a predetermined threshold; and

the second point matrix is generated by assigning a binary value to each point based on the presence or absence of pixel events at the corresponding pixel location during the acquisition period.

12 . The method of claim 9 , wherein the correlation scoring function is one of the following functions:

a cross-correlation function applied to at least one patch of adjacent points of the first and second point matrices,

a sum of absolute point-wise differences applied to at least one patch of adjacent points of the first and second point matrices, and

a sum of squared point-wise differences applied to at least one patch of adjacent points of the first and second point matrices.

13 . The method of claim 11 , wherein the correlation scoring function evaluates similarity between the first point matrix and the second point matrix using logical operations applied to patches of adjacent points in the first and second point matrices.

14 . The method of claim 4 , wherein the deblurred image is generated at multiple selected times between two successive image frames produced by the frame-based camera to provide a slow-motion video stream at a higher frame rate than a frame rate of the frame-based camera.

15 . The method of claim 4 , wherein the smoothness scoring function is one of the following functions:

a norm of 2D gradients of the comparison function,

a variance of local patches of the comparison function, and

a cross-correlation function applied to the rectified image and the rectified first matrix.

16 . A stereo camera device comprising:

a frame-based camera;

an event-based camera; and

a processor receiving and processing signals provided by the frame-based and event-based cameras, the processor configured to:

acquire an image frame from a frame-based camera using an exposure period;

rectify the image frame to generate a rectified image;

acquire a pixel event stream generated during the exposure period;

generate a first point matrix from pixel events of the pixel event stream, wherein each point of the first point matrix integrates the variation contributions of all events of the corresponding pixel during the exposure period;

rectify the first point matrix to generate a deblurring correction rectified first point matrix;

compute a comparison function by comparing the rectified image and the rectified first point matrix modified by a disparity map;

compute a smoothness scoring function applied to the comparison function while varying the disparity map;

select the disparity map that optimizes the smoothness scoring function; and

generate a deblurred image at a selected time within the exposure period by applying the selected disparity map, wherein:

the rectified first point matrix represents a deblurring correction matrix computed as a function of the selected disparity map and the selected time; and

the processor is configured to compute the deblurred image as an element-by-element ratio of the rectified image and the rectified first point matrix.

17 . The stereo camera device of claim 16 , wherein the smoothness scoring function is one of the following functions:

a norm of 2D gradients of the comparison function,

a variance of local patches of the comparison function, and

a cross-correlation function applied to the rectified image and the rectified first point matrix.

Assignments (1)

Priority Claims (1)

Continuity (1)

References Cited (13)