Granted Patent B2

US 12,293,494 · App. 17/646,684 · Granted May 6, 2025

Inventors: Tingniao Wang (Hangzhou, CN); Xiaomu Liu (Hangzhou, CN)

Assignee: ZHEJIANG DAHUA TECHNOLOGY CO., LTD.

G06T5/73G06T7/246G06T2207/10016

Abstract

Claims (107)

1. A system for video stabilization, comprising:

at least one storage device storing a set of instructions; and

at least one processor in communication with the at least one storage device, wherein when executing the set of instructions, the at least one processor is configured to stabilize a video frame-by-frame from a second frame of the video by repeating operations including:

obtaining a target frame of the video, the target frame being a current frame to be stabilized in the video;

obtaining a plurality of feature points to be tracked in the target frame;

determining whether the plurality of feature points to be tracked in the target frame include at least one feature point relating to a moving object in the video;

in response to a determination that the plurality of feature points to be tracked in the target frame include at least one feature point relating to the moving object in the video, removing the at least one feature point relating to the moving object;

estimating one or more first motion parameters based on the remaining feature points of the plurality of feature points to be tracked in the target frame, the one or more first motion parameters including a homography matrix;

performing video stabilization to the target frame based on the one or more first motion parameters;

determining whether the target frame is a last frame of the video;

in response to determining that the target frame is the last frame of the video, terminating the video stabilization;

in response to determining that the target frame is not the last frame of the video,

identifying, in the target frame after the video stabilization, at least one supplement feature point that is different from the remaining feature points of the plurality of feature points to be tracked in the target frame;

designating the at least one supplement feature point and the remaining feature points as a plurality of feature points to be tracked in a frame immediately following the target frame in the video, a count of the at least one supplement feature point being equal to a count of at least one removed feature point so that a count the plurality of feature points to be tracked in the target frame is equal to a count of the plurality of feature points to be tracked in the frame immediately following the target frame; and

initiating a new repetition of the operations.

2. The system of claim 1 , wherein to perform the video stabilization to the target frame based on the one or more first motion parameters, the at least one processor is configured to perform operations including:

performing video stabilization to the target frame by performing route planning or filtering based on the one or more first motion parameters.

3. The system of claim 1 , wherein to estimate the one or more first motion parameters based on the remaining feature points, the at least one processor is configured to perform operations including:

obtaining current coordinates of the remaining feature points in the target frame;

obtaining prior coordinates of the remaining feature points in a frame prior to the target frame in the video;

comparing the current coordinates and the prior coordinates; and

estimating the one or more first motion parameters based on a comparison result.

4. The system of claim 1 , wherein when executing the set of instructions, the at least one processor configured to perform operations further including:

determining whether the target frame is the second frame of the video.

5. The system of claim 4 , wherein the target frame is the second frame of the video; and

wherein to obtain the plurality of feature points to be tracked in the target frame, the at least one processor is configured to perform operations including:

obtaining the first frame of the video that is prior to the target frame;

determining a plurality of third feature points in the first frame; and

designating the plurality of third feature points as the plurality of feature points to be tracked in the target frame.

6. The system of claim 1 , wherein to determine whether the plurality of feature points to be tracked in the target frame include at least one feature point relating to the moving object in the video, the at least one processor is configured to perform operations including:

determining current coordinates of the plurality of feature points in the target frame;

obtaining historical coordinates of the plurality of feature points in a source frame, the plurality of feature points being determined in the source frame;

estimating one or more second motion parameters based on the current coordinates and the historical coordinates of the plurality of feature points to be tracked in the target frame;

transforming the current coordinates of the plurality of feature points based on the one or more second motion parameters;

for each of the plurality of feature points to be tracked in the target frame,

determining a difference between the historical coordinates and the transformed current coordinates of the feature point;

determining whether the difference is greater than a preset threshold; and

in response to a determination that the difference is greater than the preset threshold, designating the feature point as the feature point relating to the moving object.

7. The system of claim 6 , wherein the historical coordinates of the plurality of feature points in the source frame include coordinates that are transformed based on one or more third motion parameters associated with the source frame.

8. The system of claim 6 , wherein when executing the set of instructions, the at least one processor is configured to perform operations including:

creating a data group configured to store the historical coordinates of the plurality of feature points;

wherein to remove the at least one feature point relating to the moving object, the at least one processor is configured to perform operations including:

removing the historical coordinates of the at least one feature point relating to the moving object from the data group; and

wherein to designate the at least one supplement feature point and the remaining feature points as the plurality of feature points to be tracked in the frame immediately following the target frame in the video, the at least one processor is configured to perform operations including:

adding historical coordinates of the at least one supplement feature point to the data group, wherein the historical coordinates of the at least one supplement feature point include coordinates that are transformed based on one or more fourth motion parameters associated with the target frame.

9. The system of claim 1 , wherein to determine, in the target frame after the video stabilization, the at least one supplement feature point that is different from the remaining feature points of the plurality of feature points to be tracked in the target frame, the at least one processor is configured to perform operations including:

determining a plurality of candidate feature points in the target frame; and

randomly selecting at least one of the plurality of candidate feature points as the at least one supplement feature point.

10. The system of claim 1 , wherein when executing the set of instructions, the at least one processor is configured to perform operations including:

in response to a determination that the plurality of feature points to be tracked in the target frame include no feature point relating to the moving object in the video, performing video stabilization to the target frame based on the plurality of feature points; and

designating the plurality of feature points as the plurality of feature points to be tracked in the frame immediately following the target frame in the video.

11. A method for video stabilization implemented on a machine having one or more storage devices and one or more processors, the method is configured to stabilize a video frame-by-frame from a second frame of the video by repeating operations including:

obtaining a target frame of the video, the target frame being a current frame to be stabilized in the video;

obtaining a plurality of feature points to be tracked in the target frame;

determining whether the plurality of feature points to be tracked in the target frame include at least one feature point relating to a moving object in the video;

in response to a determination that the plurality of feature points to be tracked in the target frame include at least one feature point relating to the moving object in the video, removing the at least one feature point relating to the moving object;

estimating one or more first motion parameters based on the remaining feature points of the plurality of feature points to be tracked in the target frame, the one or more first motion parameters including a homography matrix;

performing video stabilization to the target frame based on the one or more first motion parameters;

determining whether the target frame is a last frame of the video;

in response to determining that the target frame is the last frame of the video, terminating the video stabilization;

in response to determining that the target frame is not the last frame of the video,

identifying, in the target frame after the video stabilization, at least one supplement feature point that is different from the remaining feature points of the plurality of feature points to be tracked in the target frame;

designating the at least one supplement feature point and the remaining feature points as a plurality of feature points to be tracked in a frame immediately following the target frame in the video, a count of the at least one supplement feature point being equal to a count of at least one removed feature point so that a count the plurality of feature points to be tracked in the target frame is equal to a count of the plurality of feature points to be tracked in the frame immediately following the target frame; and

initiating a new repetition of the operations.

12. The method of claim 11 , wherein the performing the video stabilization to the target frame based on the one or more first motion parameters includes:

performing video stabilization to the target frame by performing route planning or filtering based on the one or more first motion parameters.

13. The method of claim 11 , wherein the estimating the one or more first motion parameters based on the remaining feature points includes:

obtaining current coordinates of the remaining feature points in the target frame;

obtaining prior coordinates of the remaining feature points in a frame prior to the target frame in the video;

comparing the current coordinates and the prior coordinates; and

estimating the one or more first motion parameters based on a comparison result.

14. The method of claim 11 , wherein the determining whether the plurality of feature points to be tracked in the target frame include at least one feature point relating to the moving object in the video includes:

determining current coordinates of the plurality of feature points in the target frame;

obtaining historical coordinates of the plurality of feature points in a source frame, the plurality of feature points being determined in the source frame;

estimating one or more second motion parameters based on the current coordinates and the historical coordinates of the plurality of feature points to be tracked in the target frame;

transforming the current coordinates of the plurality of feature points based on the one or more second motion parameters;

for each of the plurality of feature points to be tracked in the target frame,

determining a difference between the historical coordinates and the transformed current coordinates of the feature point;

determining whether the difference is greater than a preset threshold; and

in response to a determination that the difference is greater than the preset threshold, designating the feature point as the feature point relating to the moving object.

15. The method of claim 14 , further comprising:

creating a data group configured to store the historical coordinates of the plurality of feature points;

wherein the removing the at least one feature point relating to the moving object includes:

removing the historical coordinates of the at least one feature point relating to the moving object from the data group; and

wherein the designating the at least one supplement feature point and the remaining feature points as the plurality of feature points to be tracked in the frame immediately following the target frame in the video includes:

adding historical coordinates of the at least one supplement feature point to the data group, wherein the historical coordinates of the at least one supplement feature point include coordinates that are transformed based on one or more fourth motion parameters associated with the target frame.

16. The method of claim 11 , wherein the determining, in the target frame after the video stabilization, of the at least one supplement feature point that is different from the remaining feature points of the plurality of feature points to be tracked in the target frame includes:

determining a plurality of candidate feature points in the target frame; and

randomly selecting at least one of the plurality of candidate feature points as the at least one supplement feature point.

17. A non-transitory computer readable medium, comprising at least one set of instructions for video stabilization, wherein when executed by one or more processors of a computing device, the at least one set of instructions causes the computing device to stabilize a video frame-by-frame from a second frame of the video by repeating operations including:

obtaining a target frame of the video, the target frame being a current frame to be stabilized in the video;

obtaining a plurality of feature points to be tracked in the target frame;

determining whether the plurality of feature points to be tracked in the target frame include at least one feature point relating to a moving object in the video;

in response to a determination that the plurality of feature points to be tracked in the target frame include at least one feature point relating to the moving object in the video, removing the at least one feature point relating to the moving object;

estimating one or more first motion parameters based on the remaining feature points of the plurality of feature points to be tracked in the target frame, the one or more first motion parameters including a homography matrix;

performing video stabilization to the target frame based on the one or more first motion parameters;

determining whether the target frame is a last frame of the video;

in response to determining that the target frame is the last frame of the video, terminating the video stabilization;

in response to determining that the target frame is not the last frame of the video,

identifying, in the target frame after the video stabilization, at least one supplement feature point that is different from the remaining feature points of the plurality of feature points to be tracked in the target frame;

designating the at least one supplement feature point and the remaining feature points as a plurality of feature points to be tracked in a frame immediately following the target frame in the video, a count of the at least one supplement feature point being equal to a count of at least one removed feature point so that a count the plurality of feature points to be tracked in the target frame is equal to a count of the plurality of feature points to be tracked in the frame immediately following the target frame; and

initiating a new repetition of the operations.

18. The system of claim 1 , wherein the video stabilization is performed to the frame immediately following the target frame based on historical coordinates of the at least one supplement feature point, and the historical coordinates of the at least one supplement feature point are determined based on coordinates of the one or more supplement feature points in the target frame after the video stabilization.

19. The system of claim 18 , wherein

the historical coordinates of the at least one supplement feature point include coordinates of the one or more supplement feature points in the target frame after the video stabilization; or

the historical coordinates of the at least one supplement feature point include transformed coordinates of the one or more supplement feature points that are determined by transforming the coordinates of the one or more supplement feature points in the target frame after the video stabilization using one or more second motion parameters, the one or more second motion parameters being configured to determine whether the plurality of feature points include at least one feature point relating to the moving object in the video.

20. The system of claim 1 , wherein the operations is used to stabilize, in real time, a video that is being captured, and when a image capture device is capturing a scene to generate a plurality of consecutive frames to form the video, from the second frame of the video, every time a new frame is generated, the at least one processor is configured to stabilize, in real-time, the new frame via the operations.

Assignments (1)

Priority Claims (1)

Continuity (2)

References Cited (29)