Image and video processing using multiple pipelines
An image captured using an image sensor is processed to determine control statistics and to produce an altered image which can be output for display or further processing at an image capture device, and the image is then processed according to the control statistics to produce a processed image which can be output for display or further processing at the image capture device or at another device. The altered image may have a lower resolution than the processed image. The image may be stored in a buffer. For example, the image may be retrieved from the buffer to produce the altered image, and again retrieved from the buffer to produce the processed image. The altered image may replace the image within the buffer. For example, the processed image may be produced to represent the altered image at a higher resolution.
1. An image capture device, comprising:
an image sensor configured to capture an image at a first resolution;
a memory configured to store instructions; and
a processor configured to execute the instructions to:
determine control statistics from the image;
produce an altered image representative of the image at a second resolution that is lower than the first resolution;
output the altered image for display or further processing;
produce, according to the control statistics, a processed image representative of the image at the first resolution; and
output the processed image for display or further processing.
2. The image capture device of claim 1 , further comprising a display, wherein the instructions to output the altered image for display or further processing include instructions to:
output the altered image to the display of the image capture device.
3. The image capture device of claim 2 , wherein the altered image is output to the display of the image capture device before the processed image is output for display or further processing.
4. The image capture device of claim 1 , further comprising a network interface, wherein the instructions to output the processed image for display or further processing include instructions to:
encode the processed image; and
transmit, using the network interface, the encoded processed image for decoding and display at a computing device.
5. The image capture device of claim 4 , wherein the processed image is a video frame of a video streamed from the image capture device to the computing device.
6. The image capture device of claim 1 , wherein the memory includes a buffer, and wherein the instructions include instructions to:
store the image within the buffer after the image is captured; and
retrieve the image from the buffer after the control statistics are determined.
7. A method, comprising:
obtaining an image captured using an image sensor;
using one or more first processes to determine control statistics and to produce an altered image representing the image at a first resolution;
outputting the altered image for display or further processing;
using one or more second processes to produce a processed image representing the image at a second resolution according to the control statistics; and
outputting the processed image for display or further processing.
8. The method of claim 7 , further comprising:
storing the image within a buffer after obtaining the image,
wherein using the one or more first processes comprises retrieving the image from the buffer, and wherein using the one or more second processes comprises retrieving the image from the buffer.
9. The method of claim 8 , wherein the processed image represents the altered image instead of the image, wherein using the one or more first processes comprises storing the altered image within the buffer to replace the image, and wherein using the one or more second processes comprises retrieving the altered image from the buffer.
10. The method of claim 8 , wherein using the one or more first processes comprises storing the control statistics in the buffer, and wherein using the one or more second processes comprises retrieving the control statistics from the buffer.
11. The method of claim 7 , wherein the image is a video frame of a timewarp video, wherein using the one or more first processes comprises:
scaling a timing of the video frame to achieve a desired timewarp effect.
12. The method of claim 7 , wherein the control statistics include information indicating motion elements of the image, and wherein using the one or more second processes comprises:
performing stabilization correction against the altered image according to the information indicating the motion elements of the image.
13. The method of claim 7 , wherein the image is an image of a burst image set, wherein using the one or more first processes comprises:
determining quality measures for at least some of the images of the burst image set; and
determining that the image has a highest quality measure of the quality measures.
14. The method of claim 7 , wherein outputting the altered image for display or further processing comprises:
outputting the altered image for display with zero shutter lag.
15. An image processor, comprising:
one or more first processing units configured to process an image to determine control statistics and to produce an altered image at a first resolution for display at an image capture device; and
one or more second processing units configured to process the image using the control statistics to produce a processed image at a second resolution for further processing at the image capture device, wherein the second resolution is higher than the first resolution.
16. The image processor of claim 15 , wherein, to process the image, the one or more first processing units are configured to determine whether the image is intended for display or encoding.
17. The image processor of claim 16 , wherein the one or more first processing units are further configured to:
responsive to a determination that the image is intended for encoding, perform motion estimation against at least a portion of the image.
18. The image processor of claim 15 , wherein, to process the image, the one or more second processing units are configured to modify the image according to the control statistics.
19. The image processor of claim 18 , wherein modifications to the image according to the control statistics include one or more of dead pixel correction, color lens shading, lateral chromatic processing, Bayer resizing, or scaling.
20. The image processor of claim 15 , wherein the one or more second processing units are configured to produce the processed image based on the altered image instead of the image, wherein at least one of the one or more first processing units are configured to store the altered image in a buffer after the altered image is produced using the one or more first processing units, and wherein at least one of the one or more second processing units are configured to retrieve the altered image from the buffer before the altered image is processed using the one or more second processing units.