Image processing device, image processing method, image processing program, and storage medium

1. An image processing device which sets in a frame image generated by an imaging device, a region of a size smaller than the frame image, and which corrects a position or a shape of the region according to motion of the imaging device to generate an output frame image, the image processing device comprising:

a processor; and

memory storing executable instructions that, when executed by the processor, causes the processor to perform as:

an input unit that implements sequential input of a first frame image and a second frame image;

a motion acquisition unit that acquires motion data between the first frame image and the second frame image;

a matrix operation unit that calculates a projection matrix to project the output frame image to the second frame image, from a first matrix including a rolling shutter distortion component, a second matrix including at least one of a parallel translation component in directions perpendicular to an image-shooting direction and a rotation component with respect to the image-shooting direction, and an auxiliary matrix including a motion component not included in the first matrix and the second matrix; and

a drawing unit that generates the output frame image from the second frame image by using the projection matrix,

wherein the processor further causes the matrix operation unit to perform as:

a first calculation unit that calculates the first matrix for the projection matrix by calculating the rolling shutter distortion component using a parallel translation component included in the motion data;

a second calculation unit that calculates, for each of the second frame image and frame images subsequent to the second frame image, the second matrix for the projection matrix by using the motion data, the first matrix, and a past version of the second matrix derived at a preceding operation; and

an auxiliary matrix calculation unit that, for each of the second frame image and frame images subsequent to the second frame image, calculates the auxiliary matrix for the projection matrix by using the motion data, the first matrix, and a past auxiliary matrix derived at a preceding operation.

2. The image processing device according to claim 1 , wherein the auxiliary matrix includes a component to convert a rectangle to a trapezoid.

3. The image processing device according to claim 2 , wherein the auxiliary matrix includes an enlargement/reduction component.

4. The image processing device according to claim 2 , wherein the second matrix includes an enlargement/reduction component.

5. The image processing device according to claim 2 , wherein the motion acquisition unit acquires an output value from a gyro sensor.

6. The image processing device according to claim 1 , wherein the auxiliary matrix includes an enlargement/reduction component.

7. The image processing device according to claim 6 , wherein the second matrix includes an enlargement/reduction component.

8. The image processing device according to claim 6 , wherein the motion acquisition unit acquires an output value from a gyro sensor.

9. The image processing device according to claim 1 , wherein the second matrix includes an enlargement/reduction component.

10. The image processing device according to claim 9 , wherein the motion acquisition unit acquires an output value from a gyro sensor.

11. The image processing device according to claim 1 , wherein the motion acquisition unit acquires an output value from a gyro sensor.

12. The image processing device according to claim 1 , wherein the motion acquisition unit acquires an output value from a gyro sensor.

13. An image processing method which sets in a frame image generated by an imaging device, a region of a size smaller than the frame image, and which corrects a position or a shape of the region according to motion of the imaging device to generate an output frame image, the image processing method comprising causing a processor to perform:

implementing sequential input of a first frame image and a second frame image;

acquiring motion data between the first frame image and the second frame image;

calculating a projection matrix to project the output frame image to the second frame image, from a first matrix including a rolling shutter distortion component, a second matrix including at least one of a parallel translation component in directions perpendicular to an image-shooting direction and a rotation component with respect to the image-shooting direction, and an auxiliary matrix including a motion component not included in the first matrix and the second matrix; and

generating the output frame image from the second frame image by using the projection matrix,

wherein calculating a projection matrix comprises:

calculating the first matrix for the projection matrix by calculating the rolling shutter distortion component using a parallel transition component included in the motion data;

calculating the second matrix for the projection matrix, for each of the second frame image and frame images subsequent to the second frame image, by using the motion data, the first matrix, and a past version of the second matrix derived at a preceding operation; and

calculating the auxiliary matrix for the projection matrix, for each of the second frame image and frame images subsequent to the second frame image, by using the motion data, the first matrix, and a past auxiliary matrix derived at a preceding operation.

14. A non-transitory computer-readable storage medium storing an image processing program which lets a computer function to set in a frame image generated by an imaging device, a region of a size smaller than the frame image, and to correct a position or a shape of the region according to motion of the imaging device to generate an output frame image, the image processing program letting a processor in the computer perform as:

an input unit that implements sequential input of a first frame image and a second frame image;

a motion acquisition unit that acquires motion data between the first frame image and the second frame image;

a matrix operation unit that calculates a projection matrix to project the output frame image to the second frame image, from a first matrix including a rolling shutter distortion component, a second matrix including at least one of a parallel translation component in directions perpendicular to an image-shooting direction and a rotation component with respect to the image-shooting direction, and an auxiliary matrix including a motion component not included in the first matrix and the second matrix; and

a drawing unit that generates the output frame image from the second frame image by using the projection matrix,

wherein he processor, further causes the matrix operation unit to perform as:

a first calculation unit that calculates the first matrix for the projection matrix by calculating the rolling shutter distortion component using a parallel translation component included in the motion data;

a second calculation unit that calculates, for each of the second frame image and frame images subsequent to the second frame image, the second matrix for the projection matrix by using the motion data, the first matrix, and a past version of the second matrix derived at a preceding operation; and

an auxiliary matrix calculation unit that calculates, for each of the second frame image and frame images subsequent to the second frame image, the auxiliary matrix for the projection matrix by using the motion data, the first matrix, and a past version of the auxiliary matrix derived at a preceding operation.