Real-time semantic-aware camera exposure control
An “Exposure Controller” provides various techniques for training and applying a deep convolution network to provide real-time automated camera exposure control, as a real-time function of scene semantic context, in a way that improves image quality for a wide range of image subject types in a wide range of real-world lighting conditions. The deep learning approach applied by the Exposure Controller to implement this functionality first uses supervised learning to achieve a good anchor point that mimics integral exposure control for a particular camera model or type, followed by refinement through reinforcement learning. The end-to-end system (e.g., exposure control and image capture) provided by the Exposure Controller provides real-time performance for predicting and setting camera exposure values to improve overall visual quality of the resulting image over a wide range of image capture scenarios (e.g., back-lit scenes, front lighting, rapid changes to lighting conditions, etc.).
1. A system, comprising:
a digital camera configured to capture image frames; and
a machine-learned pre-trained network (PTN) executing on the digital camera, the PTN generated by pre-training a deep convolution network by applying supervised machine learning to a set of training images and corresponding camera exposure data to mimic an exposure control system of a camera type used to capture the set of training images, such that the PTN automatically predicts camera exposure values from a current image frame captured by the digital camera as an implicit function of scene semantics extracted from the current image frame; and
wherein the digital camera is configured to apply the predicted camera exposure values to capture a next image frame.
2. The system of claim 1 wherein capture of the next image frame is delayed by an adjustable number of the captured frames after applying the predicted camera exposure values.
3. The system of claim 2 wherein the adjustable number of frames is dependent on a latency of a firmware response time of the camera.
4. The system of claim 1 further comprising bypassing existing hardware metering functions of the camera when predicting the camera exposure values.
5. A system, comprising:
a digital camera configured to capture image frames; and
a machine-learned reward network (RN) executing on the digital camera, the RN generated by training a deep convolution network on human scored image sets and corresponding camera exposure data to emulate human perception with respect to gauging image exposure quality as a function of exposure settings and semantics associated with those scored image sets, such that the RN automatically predicts camera exposure values from a current image frame captured by the digital camera as an implicit function of scene semantics extracted from the current image frame; and
wherein the digital camera is configured to apply the predicted camera exposure values to capture a next image frame.
6. The system of claim 5 wherein capture of the next image frame is delayed by an adjustable number of the captured frames after applying the predicted camera exposure values.
7. The system of claim 6 wherein the adjustable number of frames is dependent on a latency of a firmware response time of the camera.
8. The system of claim 5 further comprising bypassing existing hardware metering functions of the camera when predicting the camera exposure values.
9. A method, comprising:
capturing a set of image frames via a digital camera;
predicting, via a machine-learned pre-trained network (PTN) executing on the digital camera, camera exposure values from a current image frame captured by the digital camera as an implicit function of scene semantics extracted from the current image frame, the PTN generated by pre-training a deep convolution network by applying supervised machine learning to a set of training images and corresponding camera exposure data to mimic an exposure control system of a camera type used to capture the set of training images; and
applying the predicted camera exposure values to capture a next image frame.
10. The method of claim 9 wherein the next image frame is captured after a dynamically adjustable latency interval during which the predicted camera exposure values are applied, and wherein the dynamically adjustable latency interval is dependent on a latency of a firmware response time of the camera.
11. The method of claim 9 further comprising bypassing existing hardware metering functions of the camera when predicting the camera exposure values.