Dynamic and hierarchical image sensing and processing

1. An apparatus being part of a mobile device, comprising:

an array of pixel cells each configured to perform an image sensing operation, one or more attributes of the image sensing operation being configurable based on an image sensing configuration;

a sensor data processor configured to execute a hierarchical set of processing operations; and

a controller configured to:

receive, from the sensor data processor, a first-level processing output from a first-level processing operation of the hierarchical set of processing operations, wherein the first-level processing operation is performed on one or more first image frames captured by the array of pixel cells based on a first image sensing configuration; and

based on the first-level processing output satisfying a condition;

generate a second image sensing configuration;

control the array of pixel cells to capture one or more second image frames based on the second image sensing configuration, and

control the sensor data processor to execute a second-level processing operation on the one or more second image frames to generate a second-level processing output.

2. The apparatus of claim 1 , wherein the first-level processing operation and the second-level processing operation has different computational complexities.

3. The apparatus of claim 2 , wherein the sensor data processor is configured to execute the first-level processing operation at a lower speed than the second-level processing operation.

4. The apparatus of claim 2 , wherein the sensor data processor is configured to execute the first-level processing operation at a higher speed than the second-level processing operation.

5. The apparatus of claim 1 , wherein the first image sensing configuration selects a first subset of pixel cells of the array of pixel cells to perform the image sensing operation;

wherein the second image sensing configuration selects a second subset of pixel cells of the array of pixel cells to perform the image sensing operation; and

wherein the second subset is determined based on the first subset.

6. The apparatus of claim 1 , wherein the first image sensing configuration configures each pixel cell of the array of pixel cells to perform the image sensing operation to generate pixel data of the one or more first image frames, and to generate the one or more first image frames at a first frame rate;

wherein the second image sensing configuration configures a first subset of the array of pixel cells to perform the image sensing operation to generate pixel data of the one or more second image frames and to disable a second subset of the array of pixel cells, and to generate the one or more second image frames at a second frame rate; and

wherein the first frame rate is lower than the second frame rate.

7. The apparatus of claim 6 , wherein the first image sensing configuration further configures each pixel cell of the array of pixel cells to generate the pixel data of the one or more first image frames at a first quantization resolution;

wherein the second image sensing configuration further configures the first subset of the array of pixel cells to generate the pixel data of the one or more second image frames at a second quantization resolution; and

wherein the first quantization resolution is lower than the second quantization resolution.

8. The apparatus of claim 1 , wherein the controller is configured to update the second image sensing configuration for capturing subsequent one or more second image frames based on the second-level processing output from the one or more second image frames.

9. The apparatus of claim 1 , wherein the first-level processing output indicates first pixel locations of one or more objects belonging to a particular class;

wherein the condition comprises the one or more first image frames including pixels of a target object; and

wherein the controller is configured to, based on determining that the one or more first image frames include the pixels of the target object:

determine second pixel locations of the target object in the one or more first image frames; and

generate the second image sensing configuration to select a first subset of the array of pixel cells corresponding to the second pixel locations.

10. The apparatus of claim 9 , wherein the second-level processing operation comprises identifying the target object in the one or more second image frames; and

wherein the second-level processing output indicates a result of identifying the target object in the one or more second image frames.

11. The apparatus of claim 10 , wherein the controller is configured to update the second image sensing configuration to select a different first subset of the array of pixel cells for capturing subsequent one or more second image frames based on the result of identifying the target object in the one or more second image frames, and based on a motion model of the target object.

12. The apparatus of claim 1 , wherein the first-level processing output indicates first pixel locations of a body part;

wherein the condition comprises the body part corresponding to a hand; and

wherein the controller is configured to generate the second image sensing configuration to select a first subset of the array of pixel cells corresponding to the hand.

13. The apparatus of claim 12 , wherein the second-level processing operation comprises identifying a gesture of the hand in the one or more second image frames; and

wherein the second-level processing output indicates a result of identifying the gesture of the hand in the one or more second image frames.

14. The apparatus of claim 1 , wherein the second-level processing output indicates a first image location of a first object and a second image location of a second object in the one or more second image frames; and

wherein the sensor data processor is configured to perform a third-level processing operation of the hierarchical set of processing operations based on the first image location and the second image location to generate contextual information of a scene.

15. The apparatus of claim 14 , wherein the context information includes at least one of: an interaction between the first object and the second object, or an anomaly event.

16. The apparatus of claim 1 , wherein the first-level processing operation is performed on a map of an environment including locations of one or more physical objects and location data of the apparatus; and

wherein the condition comprises the first-level processing output indicating a particular positional relationship between the apparatus and the one or more physical objects.

17. A method being performed at a mobile device, comprising:

receiving, from a sensor data processor of the mobile device, a first-level processing output from a first-level processing operation of a hierarchical set of processing operations, wherein the first-level processing operation is performed on one or more first image frames captured by an array of pixel cells based on a first image sensing configuration;

determining, by a controller of the mobile device, that the first-level processing output satisfies a condition; and

based on the first-level processing output satisfying the condition:

generating a second image sensing configuration;

controlling the array of pixel cells to capture one or more second image frames based on the second image sensing configuration, and

controlling, by the controller, the sensor data processor to execute a second-level processing operation of the hierarchical set of processing operations on the one or more second image frames to generate a second-level processing output.

18. The method of claim 17 , wherein the first-level processing operation is performed on a map of an environment including locations of one or more physical objects and location data of the mobile device; and

wherein the condition comprises the first-level processing output indicating a particular positional relationship between the mobile device and the one or more physical objects.