Granted Patent B2

US 7,936,945 · App. 11/612,347 · Granted May 3, 2011

Abstract

Claims (80)

1. A method comprising:

computing, with a processor, a representation of a distorted object image using a mathematical model of motion skew from an image that comprises a frame in a video, the representation of the distorted object image providing an approximate shape and location within the image of the distorted object image;

removing the distorted object image from view in the image using the computed representation of the distorted object image;

replacing pixels in an area of the image where the distorted object image has been removed from view with pixels gathered from a corresponding area of a different frame in the video; and

rendering a corrected image of an object in the image using the computed representation of the distorted object image.

2. The method of claim 1 , further comprising:

creating the mathematical model of the motion skew.

3. The method of claim 2 , wherein creating the mathematical model of the motion skew comprises:

creating a second mathematical model of motion between the object and a rolling shutter image sensor at a time that the image was captured; and

using the second mathematical model of the motion of the object to create the mathematical model of the motion skew.

4. The method of claim 3 , wherein the mathematical model of the motion skew is a function of velocity of the object relative to the rolling shutter image sensor during exposure of the object, and rate of a rolling shutter action of the rolling shutter image sensor.

5. The method of claim 1 further comprising:

detecting a distorted object image in the image.

6. The method of claim 5 , wherein detecting the distorted object image further comprises:

analyzing the detected distorted object image; and

gathering information from the distorted object image;

and wherein the computing of the representation of the distorted object image using the mathematical model comprises:

using the gathered information as values for parameters in the mathematical model.

7. The method of claim 5 , wherein the image comprises a frame in a video, and wherein detecting the distorted object image further comprises:

detecting at least a portion of the distorted object image in the frame in the video;

detecting at least a portion of the distorted object image in a different frame in the video; and

computing the velocity of the object by comparing the detected object portions of the two frames;

and wherein the computing of the representation of the distorted object image using the mathematical model comprises:

using the computed velocity of the object as a value of a parameter in the mathematical model.

8. The method of claim 1 , wherein the removing of the distorted object image further comprises:

generating a mask for removing from view the distorted object image.

9. The method of claim 8 , wherein generating a mask for removing from view the distorted object image comprises using the computed representation of the distorted object image to create the mask for removing the distorted object image from view in the image.

10. The method of claim 1 , wherein the rendering of the corrected image of the object further comprises:

using the mathematical model of the motion skew to translate pixels of the distorted object image to locations in the image that form the corrected image of the object.

11. The method of claim 1 , wherein the rendering of a corrected image of the object comprises rendering the corrected image of the object to form the image of the object as the object appeared at a time proximate when the object was first exposed by a rolling shutter image sensor when capturing the image.

12. The method of claim 1 , wherein the image of the object is captured by a rolling shutter image sensor embodied in a device configured to capture video, and wherein the image comprises the frame in the video.

13. The method of claim 1 , wherein the object comprises a straight elongate object having a first end and a second end, wherein the object is moving about the first end relative to a rolling shutter image sensor during exposure of the object by the image sensor, wherein the object moves from a first angle (α) to a second angle (β) during the exposure of the object by the image sensor, wherein the method further comprises creating of the mathematical model by:

providing x- and y-coordinate axes to the image so that the y-axis is perpendicular to rows of pixels sequentially exposed by the rolling shutter image sensor;

standardizing the x- and y-coordinates so that a Euclidean distance between the first end and the second end of the object has a value of 1;

selecting a point in the image proximate the first end of the object; and

assuming that the selected point is an origin of the coordinate axes;

wherein the mathematical model of the motion skew is configured to be used to generate a representation of the distorted object image, the x-coordinate of which (x image ) is represented as a mathematical function of the y-coordinate (y) as y advances from 0 to y max , where y max is the y-coordinate of the second end of the object when the object is at angle β, the mathematical function comprising:

x image =y tan(α(y max −y)/y max +βy/y max ), where the angle α is determined by a function α=β=(θ club t shutter height clubβ ), where θ club is an angular velocity of the object during the exposure of the object, where t shutter is a time taken by the rolling shutter image sensor to expose each row of pixels before exposing a next row of pixels, where height clubβ is a height of the object in pixels in a y-direction when the object is at angle β, and where the coordinates are divided by the length of the object.

14. A computer program product comprising at least one computer-readable storage medium having computer-readable program code portions stored therein, the computer-readable program code portions comprising:

a first executable portion for computing a representation of a distorted object image based on a mathematical model of motion skew from an image that comprises a frame in a video, the representation of the distorted object image providing an approximate shape and location within an image of the distorted object image;

a second executable portion for removing the distorted object image from view in the image using the computed representation of the distorted object image, wherein the second executable portion is configured to replace pixels in an area of the image comprising the removed distorted object image with pixels gathered from a corresponding area of a different frame in the video; and

a third executable portion for rendering a corrected image of an object in the image using the computed representation of the distorted object image.

15. The computer program product of claim 14 , the computer-readable program code portions comprising:

a fourth executable portion for generating a mathematical model of the motion skew.

16. The computer program product of claim 14 , wherein the mathematical model of the motion skew is a function of velocity of the object relative to a rolling shutter image sensor during exposure of the object, and rate of a rolling shutter action of the rolling shutter image sensor.

17. The computer program product of claim 14 , the computer-readable program code portions comprising:

a fourth executable portion for detecting the distorted object image in the image.

18. The computer program product of claim 17 , wherein the fourth executable portion further comprises:

a fifth executable portion for analyzing the detected distorted object image and gathering information from the distorted object image;

and wherein the first executable portion comprises:

a sixth executable portion for computing the representation of the distorted object image from the mathematical model of the motion skew using the gathered information.

19. The computer program product of claim 17 , wherein the fourth executable portion further comprises:

a fifth executable portion for detecting at least a portion of the distorted object image in the frame in the video; detecting at least a portion of the distorted object image in a different frame in the video; and computing a velocity of the object by comparing the portions of the detected object images of the two frames;

and wherein the first executable portion further comprises:

a sixth executable portion for computing the representation of the distorted object image from the mathematical model of the motion skew using the computed velocity of the object.

20. The computer program product of claim 14 , wherein the second executable portion further comprises:

a fourth executable portion for creating a mask to remove the distorted object image from view in the image based at least partially upon the computed representation of the distorted object image.

21. The computer program product of claim 14 , wherein the third executable portion further comprises:

a fourth executable portion for translating pixels of the distorted object image to locations in the image that form the corrected image of the object using the representation of the distorted object image computed by the first executable portion.

22. An apparatus comprising:

a processor; and

a memory storing computer program code, wherein the memory and stored computer program code are configured, with the processor to cause the apparatus at least to:

compute a representation of a skewed object image based on a mathematical model of motion skew from an image that comprises a frame in a video, the representation of the skewed object image providing an approximate shape and location within the image of the skewed object image;

remove the skewed object image from view in the image using the computed representation of the skewed object image;

replace pixels in an area of the image where the skewed object image has been removed from view with pixels gathered from a corresponding area of a different frame in the video; and

render a corrected object image in the image using the representation of the skewed object image.

23. The apparatus of claim 22 , further comprising:

a rolling shutter image sensor for capturing the image; and

a display for displaying the captured image.

24. The apparatus of claim 22 , wherein the memory and stored computer program code are configured, with the processor, to cause the apparatus to geometrically translate pixels that comprise the skewed object image based on the computed representation of the skewed object image.

25. The apparatus of claim 23 , wherein the mathematical model comprises:

a mathematical function of the representation of the skewed object image as a function of a velocity of the object during exposure of the object and a rolling shutter action of a rolling shutter image sensor.

26. The apparatus of claim 25 , wherein the memory and stored computer program code are configured, with the processor, to cause the apparatus to calculate at least an approximation of a velocity of the object by comparing the frame with a different frame in the video.

27. A method comprising:

generating a mathematical model of distortion in a distorted object image, wherein generating a mathematical model comprises generating a mathematical function of a representation of the distorted object image as a function of angular velocity of an object while rotating about a point during exposure of the object and a rate of rolling shutter action of a rolling shutter image sensor, wherein the mathematical function of the distorted object image comprises:

x image =y tan(α(y max −y)/y max +βy/y max ), where x- and y-coordinates are based on x- and y-axes, where the y-axis is perpendicular to rows of pixels sequentially exposed by the rolling shutter image sensor, where the x-axis is perpendicular to the y-axis, where the coordinates are standardized so that a Euclidean distance between a first end and a second end of the object has a value of 1, where an origin of the axes is located at the first end of the object which comprises the point about which the object rotates, where x image is the x-coordinate of the representation of the distorted object image, where y max is the y-coordinate of the second end of the object when the object is at angle β, where an angle α is determined by the function α=β−(θ club t shutter height clubβ ), where θ club is the angular velocity of the object during exposure of the object, where t shutter is a time taken by the rolling shutter image sensor to expose each row of pixels before exposing a next row of pixels, and where height clubβ is a height of the distorted object image in pixels in the y-direction when the object is at angle β;

removing the distorted object image from a frame of a video by generating, with a processor, a mask for removing from view the distorted object image; and

rendering a corrected object image in the frame based on the mathematical model of the distortion.

28. The method of claim 27 , wherein the rendering of a corrected object comprises:

geometrically translating pixels that comprise the distorted object image based on the mathematical model of the distortion in the distorted object image.

Assignments (9)