Smart sensor scheduler

1. A system comprising:

an image sensor comprising a plurality of pixels that form a plurality of regions of interest (ROIs);

a plurality of image processing resources; and

a scheduler configured to perform operations comprising:

determining a priority level for a particular ROI of the plurality of ROIs based on a feature detected by one or more image processing resources of the plurality of image processing resources within initial image data associated with the particular ROI;

selecting, based on the feature detected within the initial image data, a particular image processing resource of the plurality of image processing resources by which subsequent image data generated by the particular ROI is to be processed, wherein the particular image processing resource is selected further based on the particular image processing resource being spatially co-located with the particular ROI; and

inserting, based on the priority level of the particular ROI, the subsequent image data into a processing queue of the particular image processing resource to schedule the subsequent image data for processing by the particular image processing resource.

2. The system of claim 1 , wherein the image sensor comprises a first layer of an integrated circuit, wherein the plurality of image processing resources comprise one or more additional layers of the integrated circuit, wherein image data generated by the particular ROI is configured to be processed by a first subset of image processing resources of the plurality of image processing resources, wherein the first subset of image processing resources is spatially co-located with the particular ROI on the integrated circuit, and wherein selecting the particular image processing resource comprises:

selecting the particular image processing resource from the first subset of image processing resources.

3. The system of claim 1 , wherein selecting the particular image processing resource comprises:

selecting (i) a first image processing resource of the plurality of image processing resources by which the subsequent image data generated by the particular ROI is to be processed to generate an intermediate result and (ii) a second image processing resource of the plurality of image processing resources by which the intermediate result is to be processed.

4. The system of claim 3 , wherein inserting the subsequent image data into the processing queue of the particular image processing resource comprises:

inserting the subsequent image data into a first processing queue of the first image processing resource; and

based on a position of the subsequent image data in the first processing queue, inserting the intermediate result into a second processing queue of the second image processing resource such that the intermediate result is scheduled for processing by the second image processing resource after the intermediate result is generated by the first image processing resource.

5. The system of claim 1 , wherein determining the priority level for the particular ROI comprises:

receiving, from the one or more image processing resources, an indication that the feature was detected within the initial image data; and

determining the particular ROI based on a subset of the plurality of pixels from which the one or more image processing resources obtained the initial image data.

6. The system of claim 1 , wherein determining the priority level for the particular ROI comprises:

selecting the priority level from a plurality of priority levels based on the feature detected by the one or more image processing resources, wherein each respective priority level of the plurality of priority levels is associated with at least one corresponding image data feature.

7. The system of claim 1 , wherein the particular image processing resource is selected further based on environmental conditions expected to be present at a time when the subsequent image data is generated.

8. The system of claim 1 , wherein inserting the subsequent image data into the processing queue of the particular image processing resource comprises:

determining an expiration time by which the subsequent image data is to be processed by the particular image processing resource; and

associating the expiration time with the subsequent image data in the processing queue, wherein the particular image processing resource is configured to omit processing the subsequent image data after the expiration time.

9. The system of claim 1 , wherein the initial image data is generated by the particular ROI, and wherein determining the priority level for the particular ROI comprises:

determining that the feature detected by the one or more image processing resources within the initial image data generated by the particular ROI is expected to remain within the particular ROI at a time when the subsequent image data is planned to be captured.

10. The system of claim 1 , wherein the initial image data is generated by a second ROI different from the particular ROI, and wherein determining the priority level for the particular ROI comprises:

determining that the feature detected by the one or more image processing resources within the initial image data generated by the second ROI is expected to move from the second ROI to the particular ROI and be viewable by the particular ROI at a time when the subsequent image data is planned to be captured.

11. The system of claim 1 , further comprising:

a vehicle configured to operate based on output of the plurality of image processing resources, wherein the priority level for the particular ROI is determined further based on (i) a task being carried out by the vehicle and (ii) a relationship between the task and the feature detected by the one or more image processing resources.

12. The system of claim 1 , wherein the particular image processing resource is selected further based on an amount of image data in the processing queue of the particular image processing resource.

13. The system of claim 1 , wherein the particular image processing resource is selected further based on an at least one of: (i) a first objective function that minimizes a latency between obtaining a given image data by the image sensor and processing the given image data by the plurality of image processing resources, (ii) a second objective function that maximizes a utilization of the plurality of image processing resources, or (iii) a third objective function that maximizes a throughput of image data through the plurality of image processing resources.

14. The system of claim 1 , wherein the plurality of image processing resources comprises one or more of: (i) pixel-level processing circuitry, (ii) neural network circuitry, (iii) a control system of an autonomous vehicle, or (iv) communicative connections between subsets of the plurality of image processing resources.

15. The system of claim 1 , wherein the operations further comprise:

determining to deactivate an additional ROI of the plurality of ROIs based on an additional feature detected by one or more additional image processing resources of the plurality of image processing resources within additional image data associated with the additional ROI;

selecting, based on the additional ROI, one or more further image processing resources of the plurality of image processing resources to be deactivated; and

deactivating the one or more further image processing resources.

16. A method comprising:

determining, by a scheduler, a priority level for a particular region of interest (ROI) of a plurality of ROIs formed by a plurality of pixels of an image sensor, wherein the priority level is determined based on a feature detected by one or more image processing resources of a plurality of image processing resources within initial image data associated with the particular ROI;

selecting, by the scheduler and based on the feature detected within the initial image data, a particular image processing resource of the plurality of image processing resources by which subsequent image data generated by the particular ROI is to be processed, wherein the particular image processing resource is selected further based on the particular image processing resource being spatially co-located with the particular ROI; and

inserting, by the scheduler and based on the priority level of the particular ROI, the subsequent image data into a processing queue of the particular image processing resource to schedule the subsequent image data for processing by the particular image processing resource.

17. The method of claim 16 , wherein the image sensor comprises a first layer of an integrated circuit, wherein the plurality of image processing resources comprise one or more additional layers of the integrated circuit, wherein image data generated by the particular ROI is configured to be processed by a first subset of image processing resources of the plurality of image processing resources, wherein the first subset of image processing resources is spatially co-located with the particular ROI on the integrated circuit, and wherein selecting the particular image processing resource comprises:

selecting the particular image processing resource from the first subset of image processing resources.

18. The method of claim 16 , wherein selecting the particular image processing resource comprises:

selecting (i) a first image processing resource of the plurality of image processing resources by which the subsequent image data generated by the particular ROI is to be processed to generate an intermediate result and (ii) a second image processing resource of the plurality of image processing resources by which the intermediate result is to be processed.

19. The method of claim 18 , wherein inserting the subsequent image data into the processing queue of the particular image processing resource comprises:

inserting the subsequent image data into a first processing queue of the first image processing resource; and

based on a position of the subsequent image data in the first processing queue, inserting the intermediate result into a second processing queue of the second image processing resource such that the intermediate result is scheduled for processing by the second image processing resource after the intermediate result is generated by the first image processing resource.

20. A non-transitory computer readable storage medium having stored thereon instructions that, when executed by a computing device, cause the computing device to perform operations comprising:

determining a priority level for a particular region of interest (ROI) of a plurality of ROIs formed by a plurality of pixels of an image sensor, wherein the priority level is determined based on a feature detected by one or more image processing resources of a plurality of image processing resources within initial image data associated with the particular ROI;

selecting, based on the feature detected within the initial image data, a particular image processing resource of the plurality of image processing resources by which subsequent image data generated by the particular ROI is to be processed, wherein the particular image processing resource is selected further based on the particular image processing resource being spatially co-located with the particular ROI; and

inserting, based on the priority level of the particular ROI, the subsequent image data into a processing queue of the particular image processing resource to schedule the subsequent image data for processing by the particular image processing resource.