Smart sensor implementations of region of interest operating modes
A system includes an image sensor having a plurality of pixels that form a plurality of regions of interest (ROIs), and configured to operate at a frame rate higher than a threshold rate. The system also includes an image processing resource. The system further includes control circuitry configured to perform operations that include obtaining, from the image sensor, a full-resolution image of an environment. The full-resolution image contains each respective ROI of the plurality of ROIs. The operations also include selecting a particular ROI based on the full-resolution image, and detecting an object of interest in the particular ROI. The operations include determining a mode of operation by which subsequent image data generated by the particular ROI is to be processed. The operations further include processing, based on the mode of operation and the frame rate, the image data comprising a plurality of ROI images of the object of interest.
1 . A method comprising:
obtaining, by circuitry and from an image sensor comprising a plurality of pixels that form a plurality of regions of interest (ROIs), one or more ROI images representing of an environment and generated by one or more ROIs of the plurality of ROIs;
detecting, by the circuitry and based on the one or more ROI images, an object of interest;
determining an attribute associated with the object of interest based on processing the one or more ROI images using an image processing resource of the circuitry;
selecting, by the circuitry and based on the attribute associated with the object of interest, a mode of operation from a plurality of predetermined modes of operation, wherein each respective mode of operation of the plurality of predetermined modes of operation indicates a corresponding predetermined sequence of image data types to be captured by the image sensor and processed by the image processing resource during one or more subsequent processing cycles to which the mode of operation corresponds; and
processing, by the image processing resource and based on the mode of operation, subsequent image data during the one or more subsequent processing cycles, wherein the subsequent image data comprises the corresponding predetermined sequence of the image data types of the mode of operation of the object of interest.
2 . The method of claim 1 , wherein the corresponding predetermined sequence of image data types comprises:
a plurality of ROI images and a full-resolution image;
the plurality of ROI images instead of the full-resolution image; or
the full-resolution image instead of the plurality of ROI images.
3 . The method of claim 1 , wherein the image processing resource comprises neural network circuitry, and wherein processing the subsequent image data comprises:
processing the subsequent image data using the neural network circuitry; and
generating, using the neural network circuitry, neural network output data representing results of the processing of the subsequent image data using the neural network circuitry.
4 . The method of claim 1 , wherein the image processing resource comprises pixel-level processing circuitry, and wherein processing the subsequent image data comprises:
processing the subsequent image data using the pixel-level processing circuitry; and
generating, using the pixel-level processing circuitry, processed pixel data representing results of the processing of the subsequent image data using the pixel-level processing circuitry.
5 . The method of claim 1 , wherein processing the subsequent image data comprises:
providing the subsequent image data to a server for processing thereby; and
receiving, from the server, a result of processing of the subsequent image data.
6 . The method of claim 1 , wherein processing the subsequent image data comprises performing one or more image processing tasks on the subsequent image data.
7 . The method of claim 1 , wherein the mode of operation is selected further based on a position of the image sensor on a vehicle.
8 . The method of claim 1 , wherein the image sensor is configured to obtain a full-resolution image at a first frame rate and the one or more ROI images at a second frame rate higher than the first frame rate, wherein a number of the one or more ROI images of the environment is based on one or more of the first frame rate or the second frame rate, and wherein the subsequent image data is processed based on one or more of the first frame rate or the second frame rate.
9 . The method of claim 1 , wherein:
determining the attribute associated with the object of interest comprises determining, based on the one or more ROI images, one or more of (i) a distance between the image sensor and the object of interest or (ii) a speed of the object; and
selecting the mode of operation comprises:
comparing one or more of (i) the distance to a threshold distance or (ii) the speed to a threshold speed; and
selecting the mode of operation based on results of the comparing.
10 . The method of claim 1 , wherein the subsequent image data is captured at a first time, and wherein the method further comprises:
transmitting, to a second image sensor, the subsequent image data;
selecting a second ROI from a plurality of ROIs of the second image sensor, wherein the second ROI corresponds to an expected position of the object of interest within the environment at a second time later than the first time; and
obtaining, from the second image sensor, a second plurality of ROI images from the second ROI.
11 . The method of claim 10 , wherein a first subset of the circuitry forms part of the image sensor, wherein a second subset of the circuitry forms part of the second image sensor, wherein the first subset of the circuitry is configured to: (i) obtain the one or more ROI images, (ii) detect the object of interest, and (iii) provide the expected position of the object of interest to the second image sensor, wherein the second subset of the circuitry is configured to select, based on reception of the expected position of the object of interest and based on a pose of the second image sensor with respect to the environment, the second ROI of the second image sensor and obtain the second plurality of ROI images.
12 . The method of claim 11 , wherein the first subset of the circuitry is configured to provide the expected position of the object of interest to the second image sensor by way of a peer-to-peer connection between the image sensor and the second image sensor.
13 . The method of claim 1 , further comprising:
controlling a vehicle based results of processing the subsequent image data.
14 . The method of claim 1 , wherein the detecting the object of interest comprises determining one or more of: (i) geometric properties of the object of interest, (ii) a position of the object of interest within the environment, (iii) a speed of the object of interest, (iv) an optical flow associated with the object of interest, (v) a classification of the object of interest, or (vi) a confidence value associated with detection of the object of interest.
15 . The method of claim 1 , wherein the one or more ROI images are obtained from one or more of: (i) a particular ROI expected to represent the object of interest during the one or more subsequent processing cycles or (ii) at least one ROI different from the particular ROI.
16 . The method of claim 1 , further comprising:
obtaining a full-resolution image of the environment, wherein the full-resolution image contains each respective ROI of the plurality of ROIs, and wherein the object of interest is detected further based on the full-resolution image.
17 . The method of claim 1 , wherein the circuitry forms part of the image sensor.
18 . A system comprising:
an image sensor comprising a plurality of pixels that form a plurality of regions of interest (ROIs);
an image processing resource; and
circuitry configured to perform operations comprising:
obtaining, from the image sensor, one or more ROI images of representing an environment and generated by one or more ROIs of the plurality of ROIs;
detecting, based on the one or more ROI images, an object of interest;
determining an attribute associated with the object of interest based on processing the one or more ROI images using the image processing resource;
selecting, based on the attribute associated with the object of interest, a mode of operation from a plurality of predetermined modes of operation, wherein each respective mode of operation of the plurality of predetermined modes of operation indicates a corresponding predetermined sequence of image data types to be captured by the image sensor and processed by the image processing resource during one or more subsequent processing cycles to which the mode of operation corresponds; and
based on the mode of operation, causing the image processing resource to process subsequent image data during the one or more subsequent processing cycles, wherein the subsequent image data comprises the corresponding predetermined sequence of the image data types of the mode of operation.
19 . A non-transitory computer-readable medium having stored thereon instructions that, when executed by a computing device, cause the computing device to perform operations comprising:
obtaining, from an image sensor comprising a plurality of pixels that form a plurality of regions of interest (ROIs), one or more ROI images representing an environment and generated by one or more ROIs of the plurality of ROIs;
detecting, based on the one or more ROI images, an object of interest;
determining an attribute associated with the object of interest based on processing the one or more ROI images using an image processing resource;
selecting, based on the attribute associated with the object of interest, a mode of operation from a plurality of predetermined modes of operation, wherein each respective mode of operation of the plurality of predetermined modes of operation indicates a corresponding predetermined sequence of image data types to be captured by the image sensor and processed by the image processing resource during one or more subsequent processing cycles to which the mode of operation corresponds; and
processing, based on the mode of operation, the subsequent image data during the one or more subsequent processing cycles, wherein the subsequent image data comprises the corresponding predetermined sequence of the image data types of the mode of operation.
20 . The system of claim 18 , wherein the circuitry forms part of the image sensor.