Multi-frame auto exposure control (AEC)
This disclosure provides systems, methods, and devices for image signal processing. In a first aspect, a method of image processing includes controlling an image sensor to capture image data for determining a standard dynamic range (SDR) representation of a scene, the image data comprising a first series of image frames captured with a first exposure level and a second series of image frames captured at a second exposure level. The SDR representation of the scene may be generated by determining a first output image frame based on at least a first image frame of the first series of image frames. Determining of an exposure level for the image sensor during the SDR image capture may be based on a second image frame of the second series of image frames captured by the image sensor in a multi-frame sensor configuration. Other aspects and features are also claimed and described.
1. A method, comprising:
controlling one or more image sensors to capture image data representing a standard dynamic range (SDR) representation of a scene, the image data comprising a first series of image frames captured with a first exposure level and a second series of image frames captured at a second exposure level, wherein the second exposure level is lower than the first exposure level, and wherein the controlling comprises:
determining a first output image frame based on at least a first image frame of the first series of image frames; and
determining an exposure level for the one or more image sensors based on a second image frame of the second series of image frames;
determining to capture a high dynamic range (HDR) representation of the scene; and
controlling the one or more image sensors to capture second image data based on determining to capture the HDR representation of the scene, the second image data comprising a third series of image frames captured with a third exposure level and a fourth series of image frames captured at a fourth exposure level, wherein the controlling comprises:
determining a second output image frame based on at least one image frame of the third series of image frames and at least one image frame of the fourth series of image frames.
2. The method of claim 1 , further comprising:
configuring the one or more image sensors to capture the second series of image frames at a second resolution lower than a first resolution of the first series of image frames.
3. The method of claim 2 , wherein configuring the one or more image sensors at the second resolution comprises configuring the one or more image sensors with a binning configuration.
4. The method of claim 2 , wherein configuring the one or more image sensors at the second resolution comprises configuring the one or more image sensors with a skipping configuration.
5. The method of claim 1 , wherein controlling the one or more image sensors to capture image data comprises receiving the first series of image frames from a first image sensor of the one or more image sensors and receiving the second series of image frames from a second image sensor of the one or more image sensors.
6. The method of claim 1 , wherein controlling the one or more image sensors to capture image data comprises receiving the first series of image frames from a first set of rows of a first image sensor of the one or more image sensors and receiving the second series of image frames from a second set of rows of the first image sensor different from the first set of rows.
7. The method of claim 1 , further comprising:
determining to configure the one or more image sensors with a HDR sensor configuration for controlling the one or more image sensors to capture the image data for determining the standard dynamic range (SDR) representation of a scene, wherein:
the determining is based on sensor data; and
the controlling the one or more image sensors to capture image data for determining the standard dynamic range (SDR) representation of a scene is based on the determining to configure the one or more image sensors with a HDR sensor configuration for controlling the one or more image sensors to capture image data for determining the standard dynamic range (SDR) representation of a scene.
8. The method of claim 7 , wherein the determining comprises determining the sensor data indicates a panning motion of the one or more image sensors.
9. The method of claim 7 , wherein determining to configure the one or more image sensors with the HDR sensor configuration comprises determining to configure the one or more image sensors with a multi-frame high dynamic range (MFHDR) configuration.
10. The method of claim 1 , wherein the second exposure level generates an underexposed representation of the scene.
11. The method of claim 10 , wherein values of the first output image frame are determined without reference to values of the second image frame.
12. An apparatus, comprising:
a memory storing processor-readable code; and
at least one processor coupled to the memory, the at least one processor configured to execute the processor-readable code to cause the at least one processor to perform operations including:
controlling one or more image sensors to capture image data representing a standard dynamic range (SDR) representation of a scene, the image data comprising a first series of image frames captured with a first exposure level and a second series of image frames captured at a second exposure level, wherein the second exposure level is lower than the first exposure level, and wherein the controlling comprises:
determining a first output image frame based on at least a first image frame of the first series of image frames; and
determining an exposure level for the one or more image sensors based on a second image frame of the second series of image frames
determining to capture a high dynamic range (HDR) representation of the scene; and
controlling the one or more image sensors to capture second image data based on determining to capture the HDR representation of the scene, the second image data comprising a third series of image frames captured with a third exposure level and a fourth series of image frames captured at a fourth exposure level, wherein the controlling comprises:
determining a second output image frame based on at least one image frame of the third series of image frames and at least one image frame of the fourth series of image frames.
13. The apparatus of claim 12 , wherein the processor is further configured to cause the at least one processor to perform operations including:
configuring the one or more image sensors to capture the second series of image frames at a second resolution lower than a first resolution of the first series of image frames.
14. The apparatus of claim 13 , wherein configuring the one or more image sensors at the second resolution comprises configuring the one or more image sensors with a binning configuration.
15. The apparatus of claim 13 , wherein configuring the one or more image sensors at the second resolution comprises configuring the one or more image sensors with a skipping configuration.
16. The apparatus of claim 12 , wherein controlling the one or more image sensors to capture image data comprises receiving the first series of image frames from a first image sensor of the one or more image sensors and receiving the second series of image frames from a second image sensor of the one or more image sensors.
17. The apparatus of claim 12 , wherein controlling the one or more image sensors to capture image data comprises receiving the first series of image frames from a first set of rows of a first image sensor of the one or more image sensors and receiving the second series of image frames from a second set of rows of the first image sensor different from the first set of rows.
18. The apparatus of claim 12 , wherein the processor is further configured to cause the at least one processor to perform operations including:
determining to configure the one or more image sensors with a HDR sensor configuration for controlling the one or more image sensors to capture the image data for determining the standard dynamic range (SDR) representation of a scene, wherein:
the determining is based on sensor data; and
the controlling the one or more image sensors to capture image data for determining the standard dynamic range (SDR) representation of a scene is based on the determining to configure the one or more image sensors with a HDR sensor configuration for controlling the one or more image sensors to capture image data for determining the standard dynamic range (SDR) representation of a scene.
19. The apparatus of claim 18 , wherein the determining comprises determining the sensor data indicates a panning motion of the one or more image sensors.
20. The apparatus of claim 18 , wherein determining to configure the one or more image sensors with the HDR sensor configuration comprises determining to configure the one or more image sensors with a multi-frame high dynamic range (MFHDR) configuration.
21. The apparatus of claim 12 , wherein the second exposure level generates an underexposed representation of the scene.
22. The apparatus of claim 21 , wherein values of the first output image frame are determined without reference to values of the second image frame.
23. The apparatus of claim 12 , wherein the at least one processor comprises an image signal processor coupled to the one or more image sensors.
24. A method, comprising:
controlling one or more image sensors with a first multi-frame sensor configuration to capture image data for a first representation of a scene, the image data comprising a first series of image frames captured with a first exposure level and a second series of image frames captured at a second exposure level;
determining a first output image frame of the first representation of the scene based on at least a first image frame of the first series of image frames captured while the one or more image sensors is configured with the first multi-frame sensor configuration;
determining an exposure level for the one or more image sensors based on a second image frame of the second series of image frames captured while the one or more image sensors is configured with the first multi-frame sensor configuration;
controlling the one or more image sensors to switch to a second multi-frame sensor configuration to capture second image data for determining a second representation of the scene based on a measure of motion of the one or more image sensors compared to a threshold value, the second image data comprising a third series of image frames captured with a third exposure level and a fourth series of image frames captured at a fourth exposure level; and
determining a second output image frame of the second representation of the scene based on at least a third image frame of the third series of image frames and at least a fourth image frame of the fourth series of image frames while the one or more image sensors is configured with the second multi-frame sensor configuration.
25. The method of claim 24 , wherein controlling the first image sensor with the first multi-frame sensor configuration comprises capturing the first series of image frames at a first image resolution and capturing the second series of image frames at a second image resolution lower than the first image resolution.
26. The method of claim 24 , further comprising:
controlling the one or more image sensors to switch to a third sensor configuration to capture third image data for determining a third representation of the scene, the third image data comprising a fifth series of image frames captured with a fifth exposure level; and
determining a third output image frame based on at least a fifth image frame of the fifth series of image frames based on controlling the one or more image sensors with the third sensor configuration.
27. The method of claim 26 , further comprising:
determining at least a first criteria is satisfied; and
controlling the one or more image sensors to switch from the third sensor configuration to the first multi-frame sensor configuration based on determining the first criteria is satisfied.
28. The method of claim 27 , wherein determining at least a first criteria is satisfied comprises determining a panning motion of the one or more image sensors.
29. The method of claim 24 , wherein controlling the one or more image sensors comprises receiving the first series of image frames from a first image sensor of the one or more image sensors and receiving the second series of image frames from a second image sensor of the one or more image sensors.
30. The method of claim 24 , wherein controlling the one or more image sensors comprises receiving the first series of image frames from a first set of rows of a first image sensor of the one or more image sensors and receiving the second series of image frames from a second set of rows of the first image sensor different from the first set of rows.
31. An apparatus, comprising:
a memory storing processor-readable code; and
at least one processor coupled to the memory, the at least one processor configured to execute the processor-readable code to cause the at least one processor to perform operations including:
controlling one or more image sensors with a first multi-frame sensor configuration to capture image data for determining a first representation of a scene, the image data comprising a first series of image frames captured with a first exposure level and a second series of image frames captured at a second exposure level;
determining a first output image frame of the first representation of the scene based on at least a first image frame of the first series of image frames captured while the one or more image sensors is configured with the first multi-frame sensor configuration;
determining an exposure level for the one or more image sensors based on a second image frame of the second series of image frames captured while the one or more image sensors is configured with the first multi-frame sensor configuration;
controlling the one or more image sensors to switch to a second multi-frame sensor configuration to capture second image data for determining a second representation of the scene based on a measure of motion of the one or more image sensors compared to a threshold value, the second image data comprising a third series of image frames captured with a third exposure level and a fourth series of image frames captured at a fourth exposure level; and
determining a second output image frame of the second representation of the scene based on at least a third image frame of the third series of image frames and at least a fourth image frame of the fourth series of image frames while the one or more image sensors is configured with the second multi-frame sensor configuration.
32. The apparatus of claim 31 , wherein controlling the one or more image sensors with the first multi-frame sensor configuration comprises capturing the first series of image frames at a first image resolution and capturing the second series of image frames at a second image resolution lower than the first image resolution.
33. The apparatus of claim 31 , wherein the at least one processor is further configured to perform operations including:
controlling the one or more image sensors to switch to a third sensor configuration to capture third image data for determining a third representation of a scene, the third image data comprising a fifth series of image frames captured with a first exposure level; and
determining a third output image frame based on at least a fifth image frame of the fifth series of image frames based on controlling the one or more image sensors with the third sensor configuration.
34. The apparatus of claim 33 , wherein the at least one processor is further configured to perform operations including:
determining at least a first criteria is satisfied, wherein determining at least a first criteria is satisfied comprises determining a panning motion of the one or more image sensors; and
controlling the one or more image sensors to switch from the third sensor configuration to the first multi-frame sensor configuration based on determining the first criteria is satisfied.
35. The apparatus of claim 31 , wherein controlling the one or more image sensors comprises receiving the first series of image frames from a first image sensor of the one or more image sensors and receiving the second series of image frames from a second image sensor of the one or more image sensors.
36. The apparatus of claim 31 , wherein controlling the one or more image sensors comprises receiving the first series of image frames from a first set of rows of a first image sensor of the one or more image sensors and receiving the second series of image frames from a second set of rows of the first image sensor different from the first set of rows.