IP Library Granted Patent US 11,494,881
Granted Patent B2
US 11,494,881 · App. 17/136,228 · Granted Nov 8, 2022

Global movement image stabilization systems and methods

Inventors: George H. Petkov (Heemstede, NL); Stiliyan Kalitzin (Bunnik, NL); Peter Fornell (Los Angeles, CA)
Assignee: HB Innovations, Inc.
G06T5/003G06T5/50G06T7/269G06T2207/10016G06T2207/20201
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,494,881
App. No.
17/136,228
Granted
Nov 8, 2022
Kind
B2
Abstract

A method of global movement image stabilization may include calculating a global motion group parameter with respect to first and second sequential frames for two or more elementary 2D motion components of a motion group. Each global motion group parameter may be applied to a motion group vector field that corresponds to the elementary 2D motion component to which the respective global motion group parameter applies to generate a global motion group vector field corresponding to each elementary 2D motion component of the motion group. The global motion group vector fields may be summed pointwise to generate a global motion deformation vector field that provides global motion from the second frame to the first frame. The global deformation vector field may be cumulated with a previous cumulative global deformation vector field that provides global movement from the first frame to one or more previous frames to generate a current cumulative global motion deformation vector field. The second frame may be deformed by the current cumulative global motion deformation vector field to generate a stabilized frame having global movement removed.

Claims (54)

1. A global movement image stabilization method, the method comprising:

reading a first frame and a second frame of an image sequence, wherein the first frame is chronologically earlier than the second frame in the image sequence;

calculating, with respect to motion from the second frame to the first frame, a global motion group parameter for each elementary 2D motion component of a motion group, wherein the motion group defines a group of elementary 2D motion components that decompose complex motion within a 2D plane;

applying each global motion group parameter to a motion group vector field that corresponds to the elementary 2D motion component to which the respective global motion group parameter applies to generate a global motion group vector field corresponding to each elementary 2D motion component of the motion group;

summing the global motion group vector fields pointwise to generate a global motion deformation vector field that provides global motion from the second frame to the first frame;

cumulating the global deformation vector field with a previous cumulative global deformation vector field that provides global movement from the first frame to one or more previous frames to generate a current cumulative global motion deformation vector field; and

deforming the second frame by the current cumulative global motion deformation vector field to generate a stabilized frame.

2. The method of claim 1 , wherein calculating the global motion group parameters comprises calculating a mean magnitude of motion for each elementary 2D motion component of the motion group.

3. The method of claim 1 , further comprising segmenting the first and second frames, wherein the global motion group parameters are calculated from the segmented first and second frames.

4. The method of claim 3 , wherein segmenting the first and second frames comprises applying spectral segmentation to the first and second frames using RGB values.

5. The method of claim 1 , wherein the elementary 2D motion components of the motion group comprise X translation, Y translation, dilatation, rotation, shear out, and shear in.

6. The method of claim 1 , wherein the motion group vector fields are members of a group G and are set to a frame size corresponding to the first and second frames.

7. The method of claim 6 , further comprising generating the motion group vector fields of group G.

8. The method of claim 1 , wherein the first and second frames are multichannel images.

9. The method of claim 1 , further comprising:

calculating an optical flow vector field that drives the second frame to the first frame;

decomposing the optical flow vector field by the motion group to determine a motion group decomposition vector field for each elementary 2D motion component of the motion group; and

calculating the global motion group parameters from the motion group decomposition vector fields.

10. A global movement image stabilization system, the system comprising:

a memory that stores instructions; and

processor that executes the instructions to perform operations comprising:

reading a first frame and a second frame of an image sequence, wherein the first frame is chronologically earlier than the second frame in the image sequence;

calculating, with respect to motion from the second frame to the first frame, a global motion group parameter for each elementary 2D motion component of a motion group, wherein the motion group defines a group of elementary 2D motion components that decompose complex motion within a 2D plane;

applying each global motion group parameter to a motion group vector field that corresponds to the elementary 2D motion component to which the respective global motion group parameter applies to generate a global motion group vector field corresponding to each elementary 2D motion component of the motion group;

summing the global motion group vector fields pointwise to generate a global motion deformation vector field that provides global motion from the second frame to the first frame;

cumulating the global deformation vector field with a previous cumulative global deformation vector field that provides global movement from the first frame to one or more previous frames to generate a current cumulative global motion deformation vector field; and

deforming the second frame by the current cumulative global motion deformation vector field to generate a stabilized frame.

11. The system of claim 10 , wherein the global motion group parameters comprise values for mean magnitude of motion for each elementary 2D motion component of the motion group.

12. The system of claim 10 , wherein the operations further comprise segmenting the first and second frames, and wherein the global motion group parameters are calculated from the segmented first and second frames.

13. The system of claim 12 , wherein the first and second frames are segmented by spectral segmentation using RGB values.

14. The system of claim 10 , wherein the elementary 2D motion components of the motion group comprise X translation, Y translation, dilatation, rotation, shear out, and shear in.

15. The system of claim 10 , wherein the motion group vector fields are members of a group G and are set to a frame size corresponding to the first and second frames.

16. The system of claim 15 , wherein the operations further comprise generating the motion group vector fields of group G.

17. The system of claim 10 , wherein the first and second frames are multichannel images.

18. The system of claim 10 , wherein the operations further comprise:

calculating an optical flow vector field that drives the second frame to the first frame;

decomposing the optical flow vector field by the motion group to determine a motion group decomposition vector field for each elementary 2D motion component of the motion group; and

calculating the global motion group parameters from the motion group decomposition vector fields.

19. A global movement image stabilization system, the system comprising:

a memory that stores instructions; and

processor that executes the instructions to perform operations comprising:

calculating an optical flow vector field that drives a second frame of an image sequence to a first frame of an image sequence, wherein the first frame is chronologically earlier than the second frame in the image sequence;

decomposing the optical flow vector field by a motion group to determine a motion group decomposition vector field for each elementary 2D motion component of the motion group, wherein the elementary 2D motion components decompose complex motion within a 2D plane;

calculating global motion group parameters from the motion group decomposition vector fields;

applying each global motion group parameter to a motion group vector field that corresponds to the elementary 2D motion component to which the respective global motion group parameter applies to generate a global motion group vector field corresponding to each elementary 2D motion component of the motion group;

summing the global motion group vector fields pointwise to generate a global motion deformation vector field that provides global motion from the second frame to the first frame;

cumulating the global deformation vector field with a previous cumulative global deformation vector field that provides global movement from the first frame to one or more previous frames to generate a current cumulative global motion deformation vector field; and

deforming the second frame by the current cumulative global motion deformation vector field to generate a stabilized frame.

20. The system of claim 19 , wherein the global motion group parameters comprise values for mean magnitude of motion for each elementary 2D motion component of the motion group.

21. The system of claim 19 , wherein the operations further comprise segmenting the first and second frames, and wherein the global motion group parameters are calculated from the segmented first and second frames.

22. The system of claim 21 , wherein the first and second frames are segmented by spectral segmentation using RGB values.

23. The system of claim 22 , wherein the elementary 2D motion components of the motion group comprise X translation, Y translation, dilatation, rotation, shear out, and shear in.

24. The system of claim 19 , wherein the motion group vector fields are members of a group G and are set to a frame size corresponding to the first and second frames.

25. The system of claim 24 , wherein the operations further comprise generating the motion group vector fields of group G.

Assignments (2)
SECURITY INTEREST Recorded May 18, 2022
From: HAPPIEST BABY, INC.; HB INNOVATIONS INC.
To: JPMORGAN CHASE BANK, N.A.
Reel/Frame 059945/0982 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Dec 29, 2020
From: PETKOV, GEORGE H.; KALITZIN, STILIYAN; FORNELL, PETER
To: HB INNOVATIONS, INC.
Reel/Frame 054762/0795 →