Image rotation for stream of input images
A method for processing a stream of input images is described. A stream of input images that are from an image sensor is received. The stream comprises an initial sequence of input images including a subject having an initial orientation. A change in an angular orientation of the image sensor while receiving the stream of input images is determined. In response to determining the change in the angular orientation, a subsequent sequence of input images of the stream of input images is processed for rotation to counteract the change in the angular orientation of the image sensor and maintain the subject in the initial orientation. The stream of input images is transmitted to one or more display devices.
1. A method for processing a stream of input images, the method comprising:
receiving a stream of input images from an image sensor, the stream comprising an initial sequence of input images including a subject having an initial orientation, wherein a lens associated with capturing the stream of input images causes, in captured images, lens distortion that is correctable by image processing using at least one pre-existing mesh, wherein the lens distortion is a geometric distortion and the mesh is not rotationally symmetric;
determining a change in an angular orientation of the image sensor and the lens while receiving the stream of input images;
in response to determining the change in the angular orientation, correcting a subsequent sequence of input images of the stream of input images for rotation to counteract the change in the angular orientation of the image sensor and maintain the subject in the initial orientation, including adjusting the at least one pre-existing mesh based on the change in the angular orientation; and
transmitting the stream of input images, including the corrected sequence of input images, to one or more display devices.
2. The method of claim 1 , wherein determining the change in angular orientation comprises receiving a current angular orientation from an orientation sensor associated with the image sensor.
3. The method of claim 2 , further comprising filtering the current angular orientation from the orientation sensor with a smoothing filter.
4. The method of claim 1 , wherein determining the change in angular orientation comprises tracking the subject through the stream of input images using a subject detector.
5. The method of claim 4 , further comprising determining a bounding box for the subject within a current image of the stream of input images using the subject detector.
6. The method of claim 5 , further comprising processing the current image to center the subject by cropping according to the bounding box.
7. The method of claim 1 , wherein the initial orientation is a fixed reference orientation and is one of a portrait orientation and a landscape orientation.
8. The method of claim 1 , wherein the lens distortion is rotationally symmetric with respect to geometry of captured images and correcting the subsequent sequence of input images of the stream of input images comprises performing a geometric transformation of a current image of the stream of input images based on the change in the angular orientation.
9. The method of claim 1 , wherein adjusting the mesh comprises interpolating, based on the change in the angular orientation, among a plurality of discrete meshes that correspond to the lens and correct the geometric distortion.
10. The method of claim 1 , wherein correcting the subsequent sequence of input images of the stream of input images comprises:
after adjusting the mesh, applying the adjusted mesh to a current image of the stream of input images.
11. The method of claim 1 , wherein adjusting the mesh comprises rotating the mesh based on the change in the angular orientation;
wherein correcting the subsequent sequence of input images of the stream of input images comprises applying the rotated mesh to a current image of the stream of input images.
12. A method for processing a stream of input images, the method comprising:
receiving a stream of input images from an image sensor, the stream comprising an initial sequence of input images including a subject having an initial orientation, wherein a lens associated with capturing the stream of input images causes, in captured images, lens distortion that is correctable by image processing using at least one pre-existing mesh;
determining a change in an angular orientation of the image sensor and the lens while receiving the stream of input images;
in response to determining the change in the angular orientation, correcting a subsequent sequence of input images of the stream of input images for rotation to counteract the change in the angular orientation of the image sensor and maintain the subject in the initial orientation, including adjusting the at least one pre-existing mesh based on the change in the angular orientation; and
transmitting the stream of input images, including the corrected sequence of input images, to one or more display devices;
wherein the lens distortion is a geometric distortion and the mesh is not rotationally symmetric;
wherein correcting the subsequent sequence of input images of the stream of input images for rotation comprises:
selecting the mesh from a plurality of discrete meshes based on the change in the angular orientation, each mesh of the plurality of discrete meshes being configured for a different angular orientation of the image sensor;
applying the selected mesh to a current image of the subsequent sequence of input images of the stream of input images.
13. A system for processing a stream of input images, the system comprising:
an image sensor configured to capture the stream of input images, the stream comprising an initial sequence of input images including a subject having an initial orientation;
a lens associated with capturing the stream of input images that causes, in captured images, lens distortion that is correctable by image processing using at least one pre-existing mesh, wherein the lens distortion is a geometric distortion and the mesh is not rotationally symmetric;
an orientation sensor configured to sense an angular orientation of the image sensor and the lens;
a stream processor configured to:
determine a change in an angular orientation of the image sensor while receiving the stream of input images; and
in response to determining the change in the angular orientation, correct a subsequent sequence of input images of the stream of input images for rotation to counteract the change in the angular orientation of the image sensor and maintain the subject in the initial orientation, including adjusting the at least one pre-existing mesh based on the change in angular orientation;
apply the adjusted mesh to a current image of the subsequent sequence of input images of the stream of input images;
transmit the stream of input images, including the corrected sequence of input images, to one or more display devices.
14. The system of claim 13 , wherein the stream processor is configured to center the subject within the current image.