Event signal processing method and apparatus

1. An event signal processing method of an event signal processing apparatus, the event signal processing method comprising:

receiving, by the event signal processing apparatus, from sensing elements of an event-based sensor, event signals comprising timestamps;

generating, by the event signal processing apparatus, a first timestamp map comprising pixels corresponding to the sensing elements, in which the timestamps that are received are respectively arranged;

generating, by the event signal processing apparatus, a second timestamp map by:

dividing, by the event signal processing apparatus, into sub-pixels, a target pixel among the pixels included in the first timestamp map that is generated; and

interpolating, by the event signal processing apparatus, a sub-pixel timestamp into each of the sub-pixels included in the first timestamp map, based on a target pixel timestamp of the target pixel and one or more timestamps for one or more pixels adjacent to a respective one of the sub-pixels, among the timestamps arranged respectively in the pixels; and

recognizing, by the event signal processing apparatus, a user input corresponding to the event signals, based on the second timestamp map that is generated.

2. The event signal processing method of claim 1 , wherein the interpolating, by the event signal processing apparatus, the sub-pixel timestamp into each of the sub-pixels included in the first timestamp map comprises interpolating, by the event signal processing apparatus, the sub-pixel timestamp into each of the sub-pixels included in the first timestamp map, based on an average of at least two timestamps among the timestamps arranged respectively in the pixels included in the first timestamp map.

3. The event signal processing method of claim 1 , wherein the interpolating, by the event signal processing apparatus, the sub-pixel timestamp into each of the sub-pixels included in the first timestamp map comprises setting, by the event signal processing apparatus, the sub-pixel timestamp of each of the sub-pixels to the target pixel timestamp of the target pixel.

4. The event signal processing method of claim 1 , wherein the interpolating, by the event signal processing apparatus, the sub-pixel timestamp into each of the sub-pixels included in the first timestamp map comprises:

selecting, by the event signal processing apparatus, a highest timestamp from the one or more timestamps for the one or more pixels adjacent to the respective one of the sub-pixels; and

setting, by the event signal processing apparatus, the sub-pixel timestamp of the respective one of the sub-pixels to be an average of the highest timestamp that is selected and the target pixel timestamp of the target pixel.

5. The event signal processing method of claim 1 , wherein the interpolating, by the event signal processing apparatus, the sub-pixel timestamp into each of the sub-pixels included in the first timestamp map comprises:

generating, by the event signal processing apparatus, a vector, based on the timestamps arranged respectively in the pixels included in the first timestamp map, the vector representing a flow of events corresponding to the event signals; and

interpolating, by the event signal processing apparatus, the sub-pixel timestamp into each of the sub-pixels included in the first timestamp map, based on the vector that is generated.

6. The event signal processing method of claim 1 , wherein a first resolution of the second timestamp map is higher than a second resolution of the first timestamp map.

7. The event signal processing method of claim 1 , wherein each of the sensing elements senses an event and outputs an activation signal.

8. The event signal processing method of claim 1 , wherein each of the event signals comprises identification information identifying one of the sensing elements sensing an event, and a timestamp indicating a time at which the event is sensed.

9. The event signal processing method of claim 1 , further comprising enlarging the second timestamp map that is generated,

wherein the recognizing, by the event signal processing apparatus, the user input comprises recognizing the user input corresponding to the event signals, based on the first timestamp map that is generated and the second timestamp map that is enlarged.

10. The event signal processing method of claim 1 , further comprising:

generating, by the event signal processing apparatus, a first recognition result corresponding to the event signals, based on the first timestamp map that is generated; and

generating, by the event signal processing apparatus, a second recognition result corresponding to the event signals, based on the second timestamp map that is generated,

wherein the recognizing, by the event signal processing apparatus, the user input comprises recognizing the user input corresponding to the event signals, based on the first recognition result and the second recognition result that are generated.

11. A non-transitory computer-readable storage medium storing a program comprising instructions to control a processor to perform the method of claim 1 .

12. A pixel augmentation method of a pixel augmentation apparatus based on an event-based vision sensor, the pixel augmentation method comprising:

receiving, by the pixel augmentation apparatus, time information for times at which events are sensed by sensing elements of the event-based vision sensor;

generating, by the pixel augmentation apparatus, timestamps for pixels corresponding to the sensing elements, based on the time information that is received;

dividing, by the pixel augmentation apparatus, a target pixel among the pixels into sub-pixels;

generating, by the pixel augmentation apparatus, a sub-pixel timestamp of each of the sub-pixels, based on a target pixel timestamp of the target pixel and one or more timestamps for one or more pixels adjacent to a respective one of the sub-pixels, among the timestamps for the pixels that are generated; and

recognizing, by the pixel augmentation apparatus, a user input corresponding to the events, based on the sub-pixel timestamp that is generated.

13. The pixel augmentation method of claim 12 , wherein the generating, by the pixel augmentation apparatus, the sub-pixel timestamp of each of the sub-pixels comprises setting the sub-pixel timestamp of each the sub-pixels to the target pixel timestamp of the target pixel.

14. The pixel augmentation method of claim 12 , wherein the generating, by the pixel augmentation apparatus, the sub-pixel timestamp of each of the sub-pixels comprises:

determining, by the pixel augmentation apparatus, an average of the target pixel timestamp of the target pixel and a highest timestamp among the one or more timestamps for the one or more pixels adjacent to the respective one of the sub-pixels; and

setting, by the pixel augmentation apparatus, the sub-pixel timestamp of the respective one of the sub-pixels to be the average.

15. The pixel augmentation method of claim 12 , wherein the generating, by the pixel augmentation apparatus, the sub-pixel timestamp of each of the sub-pixels comprises:

generating, by the pixel augmentation apparatus, a vector, based on the timestamps for the pixels that are generated, the vector representing a flow of the events; and

setting, by the pixel augmentation apparatus, the sub-pixel timestamp of each of the sub-pixels, based on a direction of the vector that is generated.

16. The pixel augmentation method of claim 15 , wherein the setting, by the pixel augmentation apparatus, the sub-pixel timestamp of each of the sub-pixels comprises:

selecting, by the pixel augmentation apparatus, from the sub-pixels, a first sub-pixel that is arranged perpendicular to the direction of the vector that is generated; and

setting, by the pixel augmentation apparatus, the sub-pixel timestamp of the first sub-pixel that is selected, to the target pixel timestamp of the target pixel.

17. The pixel augmentation method of claim 15 , wherein the setting the sub-pixel timestamp of each of the sub-pixels comprises:

selecting, by the pixel augmentation apparatus, from the sub-pixels, a first sub-pixel that is arranged parallel to the direction of the vector that is generated; and

generating, by the pixel augmentation apparatus, the sub-pixel timestamp of the first sub-pixel that is selected, based on an adjacent pixel timestamp of an adjacent pixel adjacent to the first sub-pixel that is selected, among the timestamps for the pixels that are generated.

18. The pixel augmentation method of claim 12 , wherein the generating, by the pixel augmentation apparatus, the sub-pixel timestamp of each of the sub-pixels comprises:

estimating, by the pixel augmentation apparatus, based on the timestamps for the pixels that are generated, a first timestamp of a first pixel corresponding to one of the sensing elements that does not sense the events;

generating, by the pixel augmentation apparatus, the sub-pixel timestamp of each of the sub-pixels, based on the first timestamp that is estimated and the timestamps for the pixels that are generated; and

restoring, by the pixel augmentation apparatus, the first timestamp of the first pixel corresponding to the one of the sensing elements, to an original timestamp.

19. An event signal processing apparatus comprising:

a processor configured to:

receive, from sensing elements of an event-based sensor, event signals comprising timestamps;

generate a first timestamp map comprising pixels corresponding to the sensing elements, in which the timestamps that are received are respectively arranged;

generate a second timestamp map by dividing, into sub-pixels, a target pixel among the pixels included in the first timestamp map that is generated, and interpolating a sub-pixel timestamp into each of the sub-pixels included in the first timestamp map, based on a target pixel timestamp of the target pixel and one or more timestamps for one or more pixels adjacent to a respective one of the sub-pixels, among the timestamps arranged respectively in the pixels; and

recognize a user input corresponding to the event signals, based on the second timestamp map that is generated.

20. The event signal processing apparatus of claim 19 , wherein the processor is further configured to:

generate a vector, based on the timestamps arranged respectively in the pixels included in the first timestamp map, the vector representing a flow of events corresponding to the event signals;

determine a pixel of the first timestamp map to be outside a boundary of the first timestamp map in a direction of the vector that is generated; and

set a timestamp of the pixel that is determined, to a null timestamp.

21. A pixel augmentation apparatus based on an event-based vision sensor, the pixel augmentation apparatus comprising:

a processor configured to:

receive time information of times at which events are sensed by sensing elements of the event-based vision sensor;

generate timestamps for pixels corresponding to the sensing elements, based on the time information that is received;

divide a target pixel among the pixels into sub-pixels;

generate a sub-pixel timestamp of each of the sub-pixels, based on a target pixel timestamp of the target pixel and one or more timestamps for one or more pixels adjacent to a respective one of the sub-pixels, among the timestamps for the pixels that are generated; and

recognize a user input corresponding to the events, based on the sub-pixel timestamp that is generated.

22. The pixel augmentation apparatus of claim 21 , wherein the processor is further configured to:

divide a sub-pixel among the sub-pixels into additional sub-pixels; and

generate a new timestamp of each of the additional sub-pixels, based on the timestamps for the pixels that are generated and the sub-pixel timestamp of each of the sub-pixels that is generated.