Vision sensor image processing device, image processing method, and program
An image processing device includes a region identifying section that identifies a first region in which a first motion occurs in a subject in an image captured by synchronous scanning, and a second region in which a second motion different from the first motion occurs in the subject or in which the subject is not substantially moving, on the basis of an event signal generated in response to a change in intensity of light in one or a plurality of pixels of the image, and an image processing section that executes image processing different between the first region and the second region on the first region and the second region, in the image.
1. An image processing device comprising:
a processor configured to:
receive, from a first vision sensor, respective frames of an image captured by the first vision sensor by synchronously scanning all pixels of a pixel array in accordance with a synchronous frame rate, wherein the respective frames of the image constitute a moving image;
identify, in each respective frame of the image captured by the first vision sensor, a first region in which a first motion occurs in which a subject substantially moves in the image captured by the first vision sensor;
receive, from a second vision sensor, event signals generated by the second vision sensor on an asynchronous basis for only respective pixels of a pixel array in which a respective intensity of light has exceeded a predetermined threshold;
identify a second region in which a second motion occurs in which the subject is not substantially moving, on a basis of event signals occurring at an edge portion of the subject detected; and
execute first image processing and second image processing on the image captured by the first vision sensor to thereby generate a plurality of output frame images, the second image processing being different from the first image processing, wherein executing the first image processing and the second image processing on the image comprises generating an output frame image of the plurality of output frame images in which a moving subject is cut out.
2. The image processing device according to claim 1 , wherein the processor is further configured to:
identify a motion direction of the subject at least in the first region, in each of the respective frame images, and
continuously execute the first image processing on the first region and the second image processing on the second region in the respective frame images, on a basis of the motion direction.
3. The image processing device according to claim 1 , wherein the processor is further configured to mark at least the first region in the image.
4. The image processing device according to claim 3 , wherein the processor is further configured to:
identify the second region in which the second motion occurs in the subject, and
mark the first region and the second region in the image in different modes.
5. The image processing device according to claim 1 , wherein the processor is further configured to mask the second region for each output frame image of the plurality of output frame images that do not include the subject.
6. The image processing device according to claim 1 , wherein the processor is further configured to:
overwrite the first region with use of the second region identified by the preceding and succeeding frame images, in each of the respective frame images.
7. An image processing method comprising:
receiving, from a first vision sensor, respective frames of an image captured by the first vision sensor by synchronously scanning all pixels of a pixel array in accordance with a synchronous frame rate, wherein the respective frames of the image constitute a moving image;
identifying, in each respective frame of the image captured by the first vision sensor, a first region in which a first motion occurs in which a subject substantially moves in an image captured by the first vision sensor;
receiving, from a second vision sensor, event signals generated by the second vision sensor on an asynchronous basis for only respective pixels of a pixel array in which a respective intensity of light has exceeded a predetermined threshold;
identifying a second region in which a second motion occurs in which the subject is not substantially moving, on a basis of event signals occurring at an edge portion of the subject detected; and
executing first image processing and second image processing on the image captured by the first vision sensor to thereby generate a plurality of output frame images the second image processing being different from the first image processing, wherein executing the first image processing and the second image processing on the image comprises generating an output frame image of the plurality of output frame images in which a moving subject is cut out.
8. A non-transitory, computer readable storage medium containing a program, which when executed by a computer, causes the computer to perform an image processing method by carrying out actions, comprising:
receiving, from a first vision sensor, respective frames of an image captured by the first vision sensor by synchronously scanning all pixels of a pixel array in accordance with a synchronous frame rate, wherein the respective frames of the image constitute a moving image;
identifying, in each respective frame of the image captured by the first vision sensor, a first region in which a first motion occurs in which a subject substantially moves in an image captured by the first vision sensor;
receiving, from a second vision sensor, event signals generated by the second vision sensor on an asynchronous basis for only respective pixels of a pixel array in which a respective intensity of light has exceeded a predetermined threshold;
identifying a second region in which a second motion occurs in which the subject is not substantially moving, on a basis of event signals occurring at an edge portion of the subject detected; and
executing first image processing and second image processing on the image captured by the first vision sensor to thereby generate a plurality of output frame images the second image processing being different from the first image processing, wherein executing the first image processing and the second image processing on the image comprises generating an output frame image of the plurality of output frame images in which a moving subject is cut out.
9. The image processing device of claim 1 , wherein each frame image of the respective frame images is associated with a respective image timestamp thereby generating a set of image timestamps and wherein each event signal is associated with respective event timestamp thereby generating a set of event timestamps.
10. The image processing device of claim 9 , wherein the set of image timestamps are synchronized with the set of event timestamps.
11. The image processing device of claim 9 , wherein the processor is further configured to:
associate the first region and the second region with a respective frame of the respective frames based on the set of image timestamps and the set of event timestamps.
12. The image processing device of claim 1 , wherein the respective frame images comprise a plurality of RGB images.
13. The image processing device of claim 1 , wherein a time associated with generating a single event signal is less than the synchronous frame rate of the first vision sensor.
14. The image processing method of claim 7 , further comprising:
calibrating, prior to synchronously scanning all pixels of the pixel array, the first vision sensor and the second vision sensor such that the event signals generated by the second vision sensor correspond to one or more pixels of the pixel array.
15. The image processing method of claim 14 , wherein calibrating the first vision sensor and the second vision sensor comprises:
displaying a calibration pattern to the first vision sensor and the second vision sensor;
flickering the calibration pattern using a light source;
capturing, responsive to flickering the calibration pattern, a plurality of calibration images using the first vision sensor and a plurality of event signals using the second vision sensor; and
adjusting one or more parameters of the first vision sensor and the second vision sensor such that the plurality of event signals correspond to the calibration images.
16. The image processing method of claim 7 , wherein each frame image of the respective frame images is associated with a respective image timestamp thereby generating a set of image timestamps and wherein each event signal is associated with respective event timestamp thereby generating a set of event timestamps.
17. The image processing method of claim 16 , wherein the set of image timestamps are synchronized with the set of event timestamps.
18. The image processing method of claim 7 , wherein a time associated with generating a single event signal is less than the synchronous frame rate of the first vision sensor.
19. The image processing method of claim 7 , further comprising:
identifying a motion direction of the subject at least in the first region, in each of the respective frame images, and
executing the first image processing on the first region and the second image processing on the second region in the respective frame images, on a basis of the motion direction.
20. The image processing method of claim 7 , wherein the second image processing comprises masking the second region for each output frame image of the plurality of output frame images that do not include the subject.