Camera systems and event-assisted image processing methods
The present disclosure provides a camera system and an event-assisted image processing method. The camera system includes an image sensor, an event-based sensor, and a processing unit. The image sensor is configured to capture visual images of a targeted scene to obtain image sensing frames with a first frequency. The event-based sensor is configured to capture event data of the targeted scene to obtain event frames with a second frequency higher than the first frequency. The processing unit is configured to: receiving the image sensing frames within a predetermined time period; accumulating the event frames within the predetermined time period; generating a temporal-spatial mask indicating interested areas for the event frames; determining geometric features in the temporal-spatial masks; synchronizing the image sensing frames and the event frames at timestamps within the predetermined time period; and fusing the temporal-spatial mask with the image sensing frames to obtain a masked visual image.
1 . A camera system, comprising:
an image sensor, configured to capture visual images of a targeted scene to obtain image sensing frames with a first frequency;
an event-based sensor, configured to capture event data of the targeted scene to obtain event frames with a second frequency higher than the first frequency;
a processing unit configured to:
receiving the image sensing frames within a predetermined time period;
accumulating the event frames within the predetermined time period;
generating a temporal-spatial mask indicating interested areas for the event frames;
determining geometric features in the temporal-spatial masks;
synchronizing the image sensing frames and the event frames at timestamps within the predetermined time period; and
fusing the temporal-spatial mask with the image sensing frames to obtain a masked visual image.
2 . The camera system of claim 1 , wherein generating the temporal-spatial mask comprises:
determining whether a scene change occurs in the targeted scene according to the event frames.
3 . The camera system of claim 1 , wherein the event frames indicate light intensity changes occurring in the targeted scene.
4 . The camera system of claim 1 , wherein the temporal-spatial mask is generated under at least one predetermined granularity.
5 . The camera system of claim 1 , further comprising a memory storing geometric shapes of specified objects,
wherein the processing unit is further configured to obtain the geometric shapes from the memory to determine the geometric features in the temporal-spatial masks.
6 . The camera system of claim 5 , wherein the predetermined shapes comprise at least one of points, straight lines, curves, circles, rectangles, triangles, ellipse, and trapezoid.
7 . The camera system of claim 5 , wherein the processing unit is further configured to encode the geometric features into a set of geometric shape parameters.
8 . The camera system of claim 7 , wherein the set of geometric shape parameters is encoded according to a shape formula corresponding to the predetermined shapes.
9 . The camera system of claim 7 , wherein the processing unit is further configured to filter out unreliable shapes.
10 . The camera system of claim 1 , wherein the image sensor is a CMOS image sensor (CIS).
11 . The camera system of claim 1 , wherein a frequency of the timestamps is substantially related to the first frequency.
12 . The camera system of claim 1 , wherein the processing unit is further configured to denoise the event frames.
13 . The camera system of claim 1 , the processing unit is further configured to analyze connected-component for the event frames.
14 . The camera system of claim 13 , wherein the processing unit is further configured to segment the event frames to into granularities.
15 . The camera system of claim 1 , the processing unit is further configured to track movements of specified objects.
16 . An event-assisted image processing method, comprising:
obtaining image sensing frames of a targeted scene with a first frequency within a predetermined time period;
obtaining event frames of the targeted scene with a second frequency higher than the first frequency within the predetermined time period;
generating a temporal-spatial mask indicating interested areas for the event frames;
determining geometric features in the temporal-spatial masks;
synchronizing the image sensing frames and the event frames at timestamps within the predetermined time period; and
fusing the temporal-spatial mask with the image sensing frames to obtain a masked visual image.
17 . The method of claim 16 , further comprising determining whether a scene change occurs in the targeted scene according to the event frames.
18 . The method of claim 16 , wherein the event frames indicate light intensity changes occurring in the targeted scene.
19 . The method of claim 16 , wherein the determining the geometric features in the temporal-spatial masks is based on stored geometric shapes of specified objects.
20 . The method of claim 19 , further comprising:
determining geometric shapes for the temporal-spatial masks based on a shape-matching algorithm; and
encoding the geometric features into a set of geometric shape parameters.