Method for the 3D reconstruction of a scene
The present invention concerns a method for the 3D reconstruction of a scene comprising the matching ( 610 ) of a first event from among the first asynchronous successive events of a first sensor with a second event from among the second asynchronous successive events of a second sensor depending on a minimisation ( 609 ) of a cost function (E). The cost function comprises at least one component from: —a luminance component (E i ) that depends at least on a first luminance signal (I u ) convoluted with a convolution core (gσ(t)), the luminance of said pixel depending on a difference between maximums (t e−,u ,t e+,u ) of said first signal; and a second luminance signal (I v ) convoluted with said convolution core, the luminance of said pixel depending on a difference between maximums (t e−,v ,t e+,v ) of said second signal; —a movement component (E M ) depending on at least time values relative to the occurrence of events located at a distance from a pixel of the first sensor and time values relative to the occurrence of events located at a distance from a pixel of the second sensor.
1. A method of 3D reconstruction of a scene, the method comprising:
receiving a first piece of asynchronous information from a first vision sensor that has a first pixel matrix positioned opposite the scene, the first piece of asynchronous information comprising, for each pixel of the first matrix, the first successive events coming from said pixel of the first matrix;
receiving a second piece of asynchronous information from a second vision sensor that has a second pixel matrix positioned opposite the scene, the second piece of asynchronous information comprising, for each pixel of the second matrix, the second successive events coming from said pixel of the second matrix, the second sensor being separate from the first sensor;
storing the first piece of asynchronous information from the first vision sensor and the second piece of asynchronous information from the second vision sensor at a memory;
matching, by a processor in communication with the memory, a first event from among the first successive events coming from the pixel of the first matrix with a second event from among the second successive events coming from the pixel of the second matrix, the second event being determined to minimize a cost function for the first event, the cost function being a simple sum or a weighted sum of a luminance component and a movement component and a time component and a geometric component for the pixel of the first matrix and the pixel of the second matrix; and
determining, by the processor, a 3D reconstruction of a scene based on the matching,
the luminance component depending on at least:
a first luminance signal coming from the pixel of the first sensor convoluted with a convolution core, the luminance of said first sensor pixel depending on a difference between the maximums of said first signal, and
a second luminance signal coming from the pixel of the second sensor convoluted with said convolution core, the luminance of said second sensor pixel depending on a difference between the maximums of said second signal,
the movement component depending on at least:
time values relating to the occurrence of events spatially located at a predetermined distance from the pixel of the first sensor, and
time values relating to the occurrence of events spatially located at a predetermined distance from the pixel of the second sensor,
the time component depending on a difference between:
a first time value relating to one of the first successive events of the first sensor, and
a second time value relating to one of the second successive events of the second sensor, and
the geometric component depending on:
a spatial distance from the pixel of the second sensor at an epipolar straight line or at an epipolar intersection defined by at least one pixel of the first sensor.
2. The method according to claim 1 , wherein the first luminance signal of the pixel of the first sensor and the second luminance signal of the pixel of the second sensor comprise a maximum, coding an occurrence time of a luminance variation, the convolution core being a predetermined Gaussian variance.
3. The method according to claim 1 , wherein said luminance component additionally depends on:
luminance signals of pixels of the first sensor, spatially located at a predetermined distance from the pixel of the first sensor, convoluted with the convolution core, and
luminance signals of pixels of the second sensor, spatially located at a predetermined distance from the pixel of the second sensor, convoluted with the convolution core.
4. The method according to claim 1 , wherein said movement component depends on:
an average value of the time values relating to the occurrence of pixel events of the first sensor, spatially located at a predetermined distance from the pixel of the first sensor, and
an average value of the time values relating to the occurrence of pixel events of the second sensor, spatially located at a predetermined distance from the pixel of the second sensor.
5. The method according to claim 1 , wherein said movement component depends on, for a given time:
for each current time value relating to the occurrence of events, spatially located at the predetermined distance from the pixel of the first sensor, of a function value decreasing from a distance of said given time to said current time value, and
for each current time value relating to the occurrence of events, spatially located at the predetermined distance from the pixel of the second sensor, of a function value decreasing from a distance of said given time to said current time value.
6. The method according to claim 1 , wherein said movement component depends on:
a first convolution of a decreasing function with a signal comprising a Dirac for each time value relating to the occurrence of events, spatially located at the predetermined distance from the pixel of the first sensor, and
a second convolution of a decreasing function with a signal comprising a Dirac for each time value relating to the occurrence of events, spatially located at the predetermined distance from the pixel of the second sensor.
7. A device for the 3D reconstruction of a scene, the device comprising:
an interface configured to
receive a first piece of asynchronous information from a first sensor that has a first pixel matrix positioned opposite the scene, the first piece of asynchronous information comprising, for each pixel of the first matrix, the first successive events coming from said pixel, and
receive a second piece of asynchronous information from a second sensor that has a second pixel matrix positioned opposite the scene, the second piece of asynchronous information comprising, for each pixel of the second matrix, the second successive events coming from said pixel, the second sensor being separate from the first sensor; and
a processor configured to
match a first event from among the first successive events coming from the pixel of the first matrix with a second event from among the second successive events coming from the pixel of the second matrix, the second event being determined to minimize a cost function for the first event, the cost function being a simple sum or a weighted sum of a luminance component and a movement component and a time component and a geometric component for the pixel of the first matrix and the pixel of the second matrix, and
determine a 3D reconstruction of a scene based on the matching,
the luminance component depending on at least:
a first luminance signal coming from the pixel of the first sensor, convoluted with a convolution core, the luminance of said first sensor pixel depending on a difference between the maximums of said first signal, and
a second luminance signal coming from the pixel of the second sensor, convoluted with said convolution core, the luminance of said second sensor pixel depending on a difference between the maximums of said second signal, and
the movement component depending on at least:
time values relating to the occurrence of events, spatially located at a predetermined distance from the pixel of the first sensor, and
time values relating to the occurrence of event, spatially located at a predetermined distance from the pixel of the second sensor,
the time component depending on a difference between:
a first time value relating to one of the first successive events of the first sensor, and
a second time value relating to one of the second successive events of the second sensor, and
the geometric component depending on:
a spatial distance from the pixel of the second sensor at an epipolar straight line or at an epipolar intersection defined by at least one pixel of the first sensor.
8. A non-transitory computer-readable storage medium comprising instructions for the implementation of the method according to claim 1 , when the method is executed by a processor.
9. The method according to claim 2 , wherein said luminance component additionally depends on:
luminance signals of pixels of the first sensor, spatially located at a predetermined distance from the pixel of the first sensor, convoluted with the convolution core, and
luminance signals of pixels of the second sensor, spatially located at a predetermined distance from the pixel of the second sensor, convoluted with the convolution core.
10. The method according to claim 2 , wherein said movement component depends on:
an average value of the time values relating to the occurrence of pixel events of the first sensor, spatially located at a predetermined distance from the pixel of the first sensor, and
an average value of the time values relating to the occurrence of pixel events of the second sensor, spatially located at a predetermined distance from the pixel of the second sensor.
11. The method according to claim 3 , wherein said movement component depends on:
an average value of the time values relating to the occurrence of pixel events of the first sensor, spatially located at a predetermined distance from the pixel of the first sensor, and
an average value of the time values relating to the occurrence of pixel events of the second sensor, spatially located at a predetermined distance from the pixel of the second sensor.