Granted Patent B2

US 12,608,906 · App. 18/572,844 · Granted Apr 21, 2026

Inventor: Hengkai Guo (Beijing, CN)

Assignee: BEIJING ZITIAO NETWORK TECHNOLOGY CO., LTD.

G06V10/44G06T7/73

Abstract

Claims (75)

1 . A plane estimation method, comprising:

obtaining multiple video frames of a target video, extracting feature points from each of the multiple video frames, determining multiple same feature points in feature points of the multiple video frames, and determining position information of the multiple same feature points in the multiple video frames;

determining a homography matrix of a plane between two adjacent video frames according to the position information of the multiple same feature points in the multiple video frames based on a random sample consensus algorithm; and

determining parameter information of the plane between the two adjacent video frames according to the homography matrix of the plane.

2 . The plane estimation method according to claim 1 , wherein the determining a homography matrix of a plane between two adjacent video frames according to the position information of multiple same feature points in the video frames based on a random sample consensus algorithm comprises:

extracting a preset number of feature points from the multiple same feature points as first feature points, and taking other feature points without being extracted as second feature points;

determining an initial homography matrix of a plane between every two adjacent video frames based on position information of the first feature points in the every two adjacent video frames;

determining whether each of the second feature points matches the initial homography matrix of the plane between the every two adjacent video frames according to position information of the each of the second feature points in the every two adjacent video frames and the initial homography matrix of the plane between the every two adjacent video frames;

for the each of the second feature points, taking the second feature point as an inlier in response to the second feature point being matched with the initial homography matrix of the plane between the every two adjacent video frames, repeating the above steps from the extracting a preset number of feature points from the multiple same features points as first feature points until a number of iterations reaches a preset number, and determining a target iteration with a maximum number of inliers among the iterations; and

determining the homography matrix of the plane between the two adjacent video frames according to position information of all inliers determined in the target iteration in the two adjacent video frames.

3 . The plane estimation method according to claim 2 , wherein the determining whether each of the second feature points matches the initial homography matrix of the plane between the every two adjacent video frames according to position information of the each of the second feature points in the every two adjacent video frames and the initial homography matrix of the plane between the every two adjacent video frames comprises:

determining re-projection position information of the each of the second feature points in a succeeding one of the every two video frames, according to the position information of the each of the second feature points in a preceding one of the every two video frames and the initial homography matrix of the plane between the every two adjacent video frames;

determining a re-projection error of the each of the second feature points between the every two adjacent video frames, according to position information and the re-projection position information of the each of the second feature points in the succeeding one of the every two video frames; and

determining whether the each of the second feature points matches the initial homography matrix of the plane between the every two adjacent video frames according to the re-projection error of the each of the second feature points between the every two adjacent video frames.

4 . The plane estimation method according to claim 3 , wherein the determining whether the each of the second feature points matches the initial homography matrix of the plane between the every two adjacent video frames according to the re-projection error of the each of the second feature points between the every two adjacent video frames comprises:

determining an average re-projection error according to the re-projection error of the each of the second feature points between the every two video frames; and

determining whether the each of the second feature points matches the initial homography matrix of the plane between the every two adjacent video frames according to the average re-projection error and a preset threshold.

5 . The plane estimation method according to claim 1 , wherein the determining a homography matrix of a plane between two adjacent video frames comprises:

after determining a homography matrix of a current plane between the two adjacent video frames, removing feature points used to determine the homography matrix of the current plane from the multiple same feature points to obtain remaining feature points; and

taking a plane other than the current plane as the current plane, determining a homography matrix of the current plane between the two adjacent video frames using the random sample consensus algorithm according to position information of the remaining feature points in the two adjacent video frames, and repeating the above steps from the removing feature points used to determine the homography matrix of the current plane from the multiple same feature points to obtain remaining feature points until a number of remaining feature points is less than a preset number.

6 . The plane estimation method according to claim 1 , further comprising: after determining the parameter information of the plane,

determining position and pose information of a virtual object according to the parameter information of the plane; and

displaying the virtual object in association with the plane in a preset video frame according to the position and pose information.

7 . An electronic device, comprising:

at least one processor;

a storage device for storing at least one program, which when executed by the at least one processor, causes the at least one processor to implement the plane estimation method according to of claim 1 .

8 . The electronic device according to claim 7 , wherein the determining a homography matrix of a plane between two adjacent video frames according to the position information of multiple same feature points in the video frames based on a random sample consensus algorithm comprises:

extracting a preset number of feature points from the multiple same feature points as first feature points, and taking other feature points without being extracted as second feature points;

determining an initial homography matrix of a plane between every two adjacent video frames based on position information of the first feature points in the every two adjacent video frames;

determining whether each of the second feature points matches the initial homography matrix of the plane between the every two adjacent video frames according to position information of the each of the second feature points in the every two adjacent video frames and the initial homography matrix of the plane between the every two adjacent video frames;

for the each of the second feature points, taking the second feature point as an inlier in response to the second feature point being matched with the initial homography matrix of the plane between the every two adjacent video frames, repeating the above steps from the extracting a preset number of feature points from the multiple same feature points as first feature points until a number of iterations reaches a preset number, and determining a target iteration with a maximum number of inliers among the iterations; and

determining the homography matrix of the plane between the two adjacent video frames according to position information of all inliers determined in the target iteration in the two adjacent video frames.

9 . The electronic device according to claim 8 , wherein the determining whether each of the second feature points matches the initial homography matrix of the plane between the every two adjacent video frames according to position information of the each of the second feature points in the every two adjacent video frames and the initial homography matrix of the plane between the every two adjacent video frames comprises:

determining re-projection position information of the each of the second feature points in a succeeding one of the every two video frames, according to the position information of the each of the second feature points in a preceding one of the every two video frames and the initial homography matrix of the plane between the every two adjacent video frames;

determining a re-projection error of the each of the second feature points between the every two adjacent video frames, according to position information and the re-projection position information of the each of the second feature points in the succeeding one of the every two video frames; and

determining whether the each of the second feature points matches the initial homography matrix of the plane between the every two adjacent video frames according to the re-projection error of the each of the second feature points between the every two adjacent video frames.

10 . The electronic device according to claim 9 , wherein the determining whether the each of the second feature points matches the initial homography matrix of the plane between the every two adjacent video frames according to the re-projection error of the each of the second feature points between the every two adjacent video frames comprises:

determining an average re-projection error according to the re-projection error of the each of the second feature points between the every two video frames; and

determining whether the each of the second feature points matches the initial homography matrix of the plane between the every two adjacent video frames according to the average re-projection error and a preset threshold.

11 . The electronic device according to claim 7 , wherein the determining a homography matrix of a plane between two adjacent video frames comprises:

after determining a homography matrix of a current plane between the two adjacent video frames, removing feature points used to determine the homography matrix of the current plane from the multiple same feature points to obtain remaining feature points; and

taking a plane other than the current plane as the current plane, determining a homography matrix of the current plane between the two adjacent video frames using the random sample consensus algorithm according to position information of the remaining feature points in the two adjacent video frames, and repeating the above steps from the removing feature points used to determine the homography matrix of the current plane from the multiple same feature points to obtain remaining feature points until a number of remaining feature points is less than a preset number.

12 . The electronic device according to claim 7 , wherein storage device further stores at least one program, which when executed by the at least one processor, causes the at least one processor to, after determining the parameter information of the plane,

determine position and pose information of a virtual object according to the parameter information of the plane; and

display the virtual object in association with the plane in a preset video frame according to the position and pose information.

13 . A non-transitory computer readable storage medium comprising computer executable instructions, which when executed by a computer processor cause the computer processor to execute the plane estimation method according to of claim 1 .

14 . The non-transitory computer readable storage medium according to claim 13 , wherein the determining a homography matrix of a plane between two adjacent video frames according to the position information of multiple same feature points in the video frames based on a random sample consensus algorithm comprises:

extracting a preset number of feature points from the multiple same feature points as first feature points, and taking other feature points without being extracted as second feature points;

determining an initial homography matrix of a plane between every two adjacent video frames based on position information of the first feature points in the every two adjacent video frames;

determining whether each of the second feature points matches the initial homography matrix of the plane between the every two adjacent video frames according to position information of the each of the second feature points in the every two adjacent video frames and the initial homography matrix of the plane between the every two adjacent video frames;

for the each of the second feature points, taking the second feature point as an inlier in response to the second feature point being matched with the initial homography matrix of the plane between the every two adjacent video frames, repeating the above steps from the extracting a preset number of feature points from the multiple same feature points as first feature points until a number of iterations reaches a preset number, and determining a target iteration with a maximum number of inliers among the iterations; and

determining the homography matrix of the plane between the two adjacent video frames according to position information of all inliers determined in the target iteration in the two adjacent video frames.

15 . The non-transitory computer readable storage medium according to claim 14 , wherein the determining whether each of the second feature points matches the initial homography matrix of the plane between the every two adjacent video frames according to position information of the each of the second feature points in the every two adjacent video frames and the initial homography matrix of the plane between the every two adjacent video frames comprises:

determining re-projection position information of the each of the second feature points in a succeeding one of the every two video frames, according to the position information of the each of the second feature points in a preceding one of the every two video frames and the initial homography matrix of the plane between the every two adjacent video frames;

determining a re-projection error of the each of the second feature points between the every two adjacent video frames, according to position information and the re-projection position information of the each of the second feature points in the succeeding one of the every two video frames; and

determining whether the each of the second feature points matches the initial homography matrix of the plane between the every two adjacent video frames according to the re-projection error of the each of the second feature points between the every two adjacent video frames.

16 . The non-transitory computer readable storage medium according to claim 15 , wherein the determining whether the each of the second feature points matches the initial homography matrix of the plane between the every two adjacent video frames according to the re-projection error of the each of the second feature points between the every two adjacent video frames comprises:

determining an average re-projection error according to the re-projection error of the each of the second feature points between the every two video frames; and

determining whether the each of the second feature points matches the initial homography matrix of the plane between the every two adjacent video frames according to the average re-projection error and a preset threshold.

17 . The non-transitory computer readable storage medium according to claim 13 , wherein the determining a homography matrix of a plane between two adjacent video frames comprises:

after determining a homography matrix of a current plane between the two adjacent video frames, removing feature points used to determine the homography matrix of the current plane from the multiple same feature points to obtain remaining feature points; and

taking a plane other than the current plane as the current plane, determining a homography matrix of the current plane between the two adjacent video frames using the random sample consensus algorithm according to position information of the remaining feature points in the two adjacent video frames, and repeating the above steps from the removing feature points used to determine the homography matrix of the current plane from the multiple same feature points to obtain remaining feature points until a number of remaining feature points is less than a preset number.

18 . The non-transitory computer readable storage medium according to claim 13 , further storing at least one program, which when executed by the at least one processor, causes the at least one processor to after determining the parameter information of the plane,

determine position and pose information of a virtual object according to the parameter information of the plane; and

display the virtual object in association with the plane in a preset video frame according to the position and pose information.

19 . A plane estimation apparatus, comprising:

a position information determination module configured to obtain multiple video frames of a target video, extract feature points from each of the multiple video frames, determine multiple same feature points in feature points of the multiple video frames, and determine position information of the multiple same feature points in the multiple video frames;

a homography matrix determination module configured to determine a homography matrix of a plane between two adjacent video frames according to the position information of the multiple same feature points in the multiple video frames based on a random sample consensus algorithm; and

a plane parameter determination module configured to determine parameter information of the plane between the two adjacent video frames according to the homography matrix of the plane.

20 . The plane estimation apparatus according to claim 19 , wherein the homography matrix determination module comprises:

an extraction unit configured to extract a preset number of feature points from the multiple same feature points as first feature points, and taking other feature points without being extracted as second feature points;

an initial matrix determination unit configured to determine an initial homography matrix of a plane between every two adjacent video frames based on position information of the first feature points in the every two adjacent video frames;

a determination unit configured to determine whether each of the second feature points matches the initial homography matrix of the plane between the every two adjacent video frames according to position information of the each of the second feature points in the every two adjacent video frames and the initial homography matrix of the plane between the every two adjacent video frames;

an inlier determination unit configured to, for the each of the second feature points, take the second feature point as an inlier in response to the second feature point being matched with the initial homography matrix of the plane between the every two adjacent video frames, repeat the above steps from the extracting a preset number of feature points from the multiple same feature points as first feature points until a number of iterations reaches a preset number, and determine a target iteration with a maximum number of inliers among the iterations; and

a final matrix determination unit configured to determine the homography matrix of the plane between the two adjacent video frames according to position information of all inliers determined in the target iteration in the two adjacent video frames.

Assignments (1)

Priority Claims (1)

Continuity (1)

References Cited (19)