Bandwidth limited context based adaptive acquisition of video frames and events for user defined tasks

1. A system to process imaging data, the system comprising:

an imaging system configured to:

capture image data of a scene and event data of the scene, wherein a red-green-blue camera imaging sensor captures the image data and an event camera imaging sensor captures the event data;

compress the image data and the event data; and

transmit the compressed image data and the compressed event data to a host; and

the host operatively coupled to the imaging system, wherein the host includes a processor configured to:

perform object detection on the compressed image data and the compressed event data to identify one or more objects;

perform object tracking on the one or more objects;

predict one or more regions of interest for subsequent data capture based on the object detection and the object tracking, wherein prediction of the one or more regions of interest includes a prediction of one or more first regions of interest based on the image data and a prediction of one or more second regions of interest based on the event data, and wherein the processor fuses the one or more first regions of interest and the one or more second regions of interest to form the one or more regions of interest; and

provide the one or more regions of interest to the imaging system to control capture of additional image data and additional event data by the imaging system.

2. The system of claim 1 , wherein the imaging system compresses the image data and the event data with quadtree compression.

3. The system of claim 2 , wherein the imaging system performs a quadtree decomposition of the image data and the event data to form quadtree blocks as part of the quadtree compression.

4. The system of claim 3 , wherein the imaging system is configured to spatially sample the event data based at least in part on a size of the quadtree blocks to provide spatial redundancy of the event data.

5. The system of claim 4 , wherein the imaging system is configured to discretize timestamps corresponding to the event data to provide temporal redundancy of the event data.

6. The system of claim 2 , wherein the imaging system is configured to minimize a weighted rate distortion of the event data to perform the quadtree decomposition.

7. The system of claim 6 , wherein the weighted rate distortion is normalized over an area that includes at least a portion of the one or more regions of interest.

8. The system of claim 6 , wherein the imaging system uses a dynamic Viterbi algorithm to minimize the weighted rate distortion of the event data.

9. The system of claim 3 , wherein each of the quadtree blocks includes one or more skip modes, one or more acquire modes, and a value for each of the one or more acquire modes.

10. The system of claim 1 , wherein the image data comprises distorted intensity data, and wherein the object detection is performed through a faster region-based convolutional neural network.

11. The system of claim 1 , wherein the object tracking is performed with a Kalman filter based object tracker.

12. The system of claim 1 , wherein the processor is further configured to perform edge reconstruction to correct distorted edges of the one or more objects.

13. The system of claim 1 , wherein the one or more regions of interest provided to the imaging system are in the form of one or more bounding box locations.

14. A method of processing imaging data, the method comprising:

capturing, by an imaging system that includes a red-green-blue camera imaging sensor and an event camera imaging sensor, image data of a scene and event data of the scene, wherein the red-green-blue camera imaging sensor captures the image data and the event camera imaging sensor captures the event data;

compressing, by the imaging system, the image data and the event data;

transmitting, by the imaging system, the compressed image data and the compressed event data to a host;

performing, by a processor of the host, object detection on the compressed image data and the compressed event data to identify one or more objects;

performing, by the processor, object tracking on the one or more objects;

predicting one or more regions of interest for subsequent data capture based on the object detection and the object tracking, wherein predicting the one or more regions of interest includes predicting one or more first regions of interest based on the image data and predicting one or more second regions of interest based on the event data, and wherein the processor performs fusing of the one or more first regions of interest and the one or more second regions of interest to form the one or more regions of interest; and

providing the one or more regions of interest to the imaging system to control capture of additional image data and additional event data by the imaging system.

15. The method of claim 14 , wherein the compressing comprises performing a quadtree decomposition of the image data and the event data to form quadtree blocks.

16. The method of claim 15 , further comprising spatially sampling the event data based at least in part on a size of the quadtree blocks to provide spatial redundancy of the event data.

17. The method of claim 15 , further comprising minimizing a weighted rate distortion of the event data to perform the quadtree decomposition.

18. The method of claim 14 , further comprising discretizing, by the imaging system, timestamps corresponding to the event data to provide temporal redundancy of the event data.