IP Library › Granted Patent US 10,992,887
Granted Patent B2
US 10,992,887 · App. 16/651,215 · Granted Apr 27, 2021

System and method for event camera data processing

Inventor: Aleksandr M. Movshovich (Santa Clara, CA)
Assignee: APPLE INC.
H04N5/347H04N5/335H04N5/374
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,992,887
App. No.
16/651,215
Granted
Apr 27, 2021
Kind
B2
Abstract

In some embodiments, a method is performed at a device with a processor, non-transitory memory, and an event camera including pixel sensors distributed across an area. The method includes converting an event stream from a pixel sensor over a first time period into event frames by dividing the first time period into sub-periods, and binning pixel events of the event stream, where each of the sub-periods is associated with a frame sub-period identifier. The method further includes addressing the pixel sensors by sub-dividing the area into tiles, where each of the tiles includes a grouping of the pixel sensors, and a tile address of a particular pixel sensor is a combination of a tile identifier and a position locator of the particular pixel sensor. The method further includes encoding the pixel events as a function of a tile address, a frame sub-period identifier, and a brightness indicator value.

Claims (37)

1. A method comprising:

at a device with one or more processors, non-transitory memory, and an event camera including a plurality of pixel sensors distributed across a first area:

converting an event stream from a pixel sensor over a first time period into event frames by dividing the first time period into a plurality of sub-periods, and binning pixel events of the event stream based on the plurality of sub-periods and respective timestamps of the pixel events, wherein each of the plurality of sub-periods is associated with a corresponding frame sub-period identifier;

addressing the plurality of pixel sensors by sub-dividing the first area into tiles, wherein each of the tiles includes a respective grouping of the plurality of pixel sensors, wherein a tile address of a particular pixel sensor is a combination of a respective tile identifier and a position locator of the particular pixel sensor associated with the respective tile identifier; and

encoding the pixel events as a function of a respective tile address, a frame sub-period identifier, and a brightness indicator value.

2. The method of claim 1 , wherein the brightness indicator value includes one of a number of discrete threshold level values.

3. The method of claim 1 , wherein binning the pixel events includes, within a particular one of the plurality of sub-periods, overwriting older pixel events with newer pixel events.

4. The method of claim 1 , wherein the tiles are equally sized and uniformly distributed across the first area.

5. The method of claim 1 , wherein each of a first group of one or more of the tiles has an area larger than each of a second group of one or more tiles.

6. The method of claim 1 , wherein the position locator of the particular pixel sensor includes two-dimensional coordinates of the particular pixel sensor.

7. The method of claim 1 , wherein addressing the plurality of pixel sensors includes generating a frame header for a particular event frame by associating a respective frame sub-period identifier with the particular event frame.

8. The method of claim 1 , wherein addressing the plurality of pixel sensors includes generating a tile header for a particular tile by associating a respective tile identifier with the particular tile.

9. The method of claim 1 , further comprising:

obtaining a respective event stream from each of the plurality of pixel sensors over the first time period; and

binning pixel events for each event stream.

10. The method of claim 1 , wherein converting the event stream includes converting the event stream created in accordance with a first protocol to a second protocol, different from the first protocol.

11. The method of claim 1 , wherein encoding the pixel events as the function of the respective tile address, the frame sub-period identifier, and the brightness indicator value includes:

determining that a number of pixel events within a respective tile associated with the respective tile address breaches a threshold volume within a sub-period associated with the sub-period identifier; and

encoding, in accordance with a first encoding process, the pixel events in the respective tile in response to determining breach of the threshold volume.

12. The method of claim 1 , wherein encoding the pixel events as the function of the respective tile address, the frame sub-period identifier, and the brightness indicator value includes:

determining that a number of pixel events within a respective tile associated with the respective tile address does not breach a threshold volume within a sub-period associated with the sub-period identifier; and

encoding, in accordance with a second encoding process, the pixel events in the respective tile in response to determining no breach of the threshold volume.

13. A device comprising:

an event camera including a plurality of pixel sensors distributed across a first area;

an event framer to convert a respective event stream, outputted from a respective one of the plurality of pixel sensors during a first time period, into event frames by dividing the first time period into a plurality of sub-periods and binning pixel events in the respective event stream based on the plurality of sub-periods and respective timestamps of the pixel events;

a tiler to address the plurality of pixel sensors by sub-dividing the first area into tiles, wherein each of the tiles includes a grouping of the plurality of pixel sensors, wherein a tile address of a particular pixel sensor is a combination of a respective tile identifier and a position locator of the particular pixel sensor associated with the respective tile identifier; and

an encoder to encode the pixel events as a function of a respective tile address, a frame sub-period identifier, and a brightness indicator value.

14. The device of claim 13 , wherein the event framer overwrites older pixel events with newer pixel events.

15. The device of claim 13 , wherein each of a first group of one or more of the tiles has an area larger than each of a second group of one or more tiles.

16. The device of claim 13 , wherein the tiler further generates a frame header for a particular event frame by associating a respective frame sub-period identifier with the particular event frame.

17. The device of claim 13 , wherein the tiler further generates a tile header for a particular tile by associating a respective tile identifier with the particular tile.

18. The device of claim 13 , wherein the encoder generates a code sequence individually representing a respective brightness indicator value of a pixel corresponding to a respective pixel sensor in the first area, in response to a determination that a number of pixel events within a respective tile associated with the respective tile address breaches a threshold volume within a sub-period associated with the sub-period identifier.

19. The device of claim 13 , wherein the encoder determines whether to use a first encoding process or a second encoding process based on a threshold volume of a particular tile.

20. A non-transitory computer readable storage medium has stored therein instructions, which, when executed by one or more processors of an image processing device with an event camera including a plurality of pixel sensors distributed across a first area, cause the image processing device to:

convert an event stream from a pixel sensor over a first time period into event frames by dividing the first time period into a plurality of sub-periods, and bin pixel events of the event stream based on the plurality of sub-periods and respective timestamps of the pixel events, wherein each of the plurality of sub-periods is associated with a corresponding frame sub-period identifier;

address the plurality of pixel sensors by sub-dividing the first area into tiles, wherein each of the tiles includes a respective grouping of the plurality of pixel sensors, wherein a tile address of a particular pixel sensor is a combination of a respective tile identifier and a position locator of the particular pixel sensor associated with the respective tile identifier; and

encode the pixel events as a function of a respective tile address, a frame sub-period identifier, and a brightness indicator value.

Continuity (2)
Provisional Application 62564818 · Sep 28, 2017
Related Publication 20200260022A1 · Aug 13, 2020
Cited By (1)
US 12,258,025