Image capture and desired timing image reconstruction device, system, method, and non-transitory computer readable medium

1. An information processing device comprising:

a memory storing a program, and

at least one processor configured to execute the program to perform operations comprising:

acquiring information including event data and a detection time of the event data associated with each other, the event data being data output from a first sensor including a pixel array unit, the pixel array unit having a plurality of pixels arranged in a two-dimensional lattice pattern, each of the pixels configured to detect a luminance change in incident light and output event data;

recognizing an object detected by the first sensor by using a trained model using information related to the detection time and the event data as an input; and

outputting an image related to the recognized object,

acquiring frame image data from a second sensor that captures a frame image; and

generating a reconstructed image of a desired timing obtained by combining an illustrative image of the object with the frame image data.

2. The information processing device according to claim 1 , wherein the operations further comprise:

setting, as the input to the trained model, an input image in which information indicating whether luminance included in the event data has changed to positive or negative with respect to a threshold is set as color information, and

setting, as luminance information, difference information between a reference time and the detection time.

3. The information processing device according to claim 1 , wherein the object image is an image in which information indicating whether luminance included in the event data has changed to positive or negative with respect to a threshold is set as color information and in which difference information between a reference time and the detection time is set as luminance information.

4. The information processing device according to claim 1 , wherein the operations further comprise:

detecting a timing at which the object is positioned at a predetermined location based on movement information of the object, and

outputting the image related to the object at the timing.

5. The information processing device according to claim 4 , wherein the operations further comprise:

outputting a plurality of frame images in a period including the timing, the plurality of frame images including the image related to the object.

6. The information processing device according to claim 5 , wherein the operations further comprise:

outputting the plurality of frame images having different frame rates.

7. An information processing system comprising:

a first sensor including a pixel array unit, the pixel array unit having a plurality of pixels arranged in a two-dimensional lattice pattern, each of the pixels configured to detect a luminance change in incident light and output event data; and

an information processing device,

wherein the information processing device includes:

a memory storing a program, and

at least one processor configured to execute the program to perform operations comprising:

acquiring information including the event data output from the first sensor and a detection time of the event data associated with each other;

recognizing an object detected by the first sensor by using a trained model using information related to the detection time and the event data as an input; and

outputting an image related to the recognized object,

wherein acquiring information including the event data output from the first sensor and a detection time of the event data associated with each other includes acquiring frame image data from a second sensor that captures a frame image, and

outputting an image related to the recognized object comprises generating a reconstructed image of a desired timing obtained by combining an illustrative image of the object with the frame image data.

8. An information processing method comprising:

acquiring information including event data and a detection time of the event data associated with each other, the event data being data output from a first sensor including a pixel array unit, the pixel array unit having a plurality of pixels arranged in a two-dimensional lattice pattern, each of the pixels configured to detect a luminance change in incident light and output event data;

acquiring frame image data from a second sensor that captures a frame image, recognizing an object detected by the first sensor by using a trained model using information related to the detection time and the event data as an input; and

outputting an image related to the recognized object,

wherein the image related to the recognized object is a reconstructed image of a desired timing obtained by combining an illustrative image of the object with the frame image data.

9. The method of claim 8 , wherein the method further comprises:

setting, as the input to the trained model, an input image in which information indicating whether luminance included in the event data has changed to positive or negative with respect to a threshold is set as color information, and

setting, as luminance information, difference information between a reference time and the detection time.

10. The method of claim 8 , wherein the method further comprises:

detecting a timing at which the object is positioned at a predetermined location based on movement information of the object, and

outputting the image related to the object at the timing.

11. The method of claim 10 , wherein the method further comprises:

outputting a plurality of frame images in a period including the timing, the plurality of frame images including the image related to the object.

12. The method of claim 11 , wherein the method further comprises:

outputting the plurality of frame images having different frame rates.

13. A non-transitory computer readable medium storing a program for causing an information processing device to perform a set of operations comprising:

acquiring information including event data and a detection time of the event data associated with each other, the event data being data output from a first sensor including a pixel array unit, the pixel array unit having a plurality of pixels arranged in a two-dimensional lattice pattern, each of the pixels configured to detect a luminance change in incident light and output event data;

recognizing an object detected by the first sensor by using a trained model using information related to the detection time and the event data as an input; and

outputting an image related to the recognized object,

wherein acquiring information including event data and a detection time of the event data associated with each other comprises acquiring frame image data from a second sensor that captures a frame image, and

wherein outputting an image related to the recognized object comprises generating a reconstructed image of a desired timing obtained by combining an illustrative image of the object with the frame image data.

14. The non-transitory computer readable medium of claim 13 , wherein the operations further comprise:

setting, as the input to the trained model, an input image in which information indicating whether luminance included in the event data has changed to positive or negative with respect to a threshold is set as color information, and

setting, as luminance information, difference information between a reference time and the detection time.

15. The non-transitory computer readable medium of claim 13 , wherein the operations further comprise:

detecting a timing at which the object is positioned at a predetermined location based on movement information of the object, and

outputting the image related to the object at the timing.

16. The non-transitory computer readable medium of claim 15 , wherein the operations further comprise:

outputting a plurality of frame images in a period including the timing, the plurality of frame images including the image related to the object.

17. The non-transitory computer readable medium of claim 16 , wherein the operations further comprise:

outputting the plurality of frame images having different frame rates.