Relocalization method and apparatus in camera pose tracking process, device, and storage medium

1. A relocalization method in a camera pose tracking process for a device, comprising:

obtaining, by a front-end program run on a device, a target image acquired after an i th marker image in the plurality of marker images, i being a positive integer, the front-end program responsible for sequentially performing the camera pose tracking process on a plurality of marker images;

determining, by the front-end program, the target image as an (i+1) th marker image upon determining that the target image satisfies a relocalization condition;

transmitting, by the front-end program, the target image to a back-end program run on the device;

performing obtaining, by the front-end program, feature point tracking on a current image acquired after the target image relative to the target image to obtain a first pose parameter, the first pose parameter representing a camera pose change from the target image to the current image;

performing, by the back-end program, relocalization on the target image to obtain a second pose parameter, and transmitting the second pose parameter to the front-end program, the second pose parameter representing the camera pose change from the first marker image to the target image; and

calculating, by the front-end program, a current pose parameter of the current image according to the first pose parameter and the second pose parameter.

2. The method according to claim 1 , wherein the performing, by the back-end program, relocalization on the target image to obtain a second pose parameter comprises at least one of:

performing, by the back-end program, relocalization on the target image relative to the first marker image, to obtain the second pose parameter;

or,

performing, by the back-end program, relocalization on the target image relative to a keyframe in a keyframe database, to obtain the second pose parameter, the keyframe being an image that has a successful relocalization result relative to the first marker image.

3. The method according to claim 2 , wherein the keyframe database comprises at least one of:

a first-order keyframe with successful first-order relocalization relative to the first marker image;

or,

an nth-order keyframe with successful nth-order relocalization relative to the (n−1) th -order keyframe in the keyframe database, n being an integer greater than 1.

4. The method according to claim 1 , wherein the calculating, by the front-end program, a current pose parameter of the current image according to the first pose parameter and the second pose parameter comprises:

calculating a current pose parameter comprising R n and T n according to the following formula:

[

R

n

T

n

0

1

]

=

[

R

ca

s

*

T

ca

0

1

]

*

[

R

relocalize

T

relocalize

0

1

]

,

the first pose parameter comprising a first rotation matrix R ca and a first displacement vector T ca , the second pose parameter comprising a second rotation matrix R relocalize and a second displacement vector T relocalice , and s being a scale corresponding to the target image.

5. The method according to claim 4 , further comprising:

mapping, by the front-end program, an initial feature point in the first marker image onto the target image to obtain a projected feature point, and calculating the scale corresponding to the target image according to an average depth of the projected feature point.

6. The method according to claim 1 , wherein the transmitting, by the front-end program, the target image to the back-end program comprises:

writing, by the front-end program, the target image into a relocalization image list in a result storage space shared by the front-end program and the back-end program; and

reading, by the back-end program, the target image from the relocalization image list.

7. The method according to claim 6 , wherein the writing, by the front-end program, the target image into a relocalization image list comprises:

detecting, by the front-end program, whether a result status bit in the result storage space is a ready state; and

writing, by the front-end program, the target image into the relocalization image list in the storage space upon detecting the result status bit is the ready state.

8. The method according to claim 6 , further comprising:

writing, by the front-end program, the target image into an expand-map list in the result storage space; and

reading, by the back-end program, the target image from the expand-map list upon determining that the target image has successful relocalization and satisfies a preset condition, and adding the target image to the keyframe database.

9. The method according to claim 1 , wherein the transmitting, by the back-end program, the second pose parameter to the front-end program comprises:

writing, by the back-end program, the second pose parameter into a result storage location in a result storage space shared by the front-end program and the back-end program; and

reading, by the front-end program, the second pose parameter from the result storage location.

10. A relocalization apparatus, comprising: a memory; and one or more processors coupled to the memory and configured to execute a front-end program and a back-end program, wherein the front-end program is configured to:

sequentially perform a camera pose tracking process on a plurality of marker images;

obtain a target image acquired after an i th marker image in the plurality of marker images, i being a positive integer;

determine the target image as an (i+1) th marker image upon determining that the target image satisfies a relocalization condition, and transmit the target image to the back-end program, the back-end program module being configured to: perform relocalization on the target image to obtain a second pose parameter, and transmit the second pose parameter to the front-end program, the second pose parameter representing a camera pose change from the first marker image to the target image;

perform feature point tracking on a current image acquired after the target image relative to the target image to obtain a first pose parameter, the first pose parameter representing a camera pose change from the target image to the current image; and

calculate a current pose parameter of the current image according to the first pose parameter and the second pose parameter.

11. The apparatus according to claim 10 , wherein

the back-end program is configured to perform at least one of: relocalization on the target image relative to the first marker image, to obtain the second pose parameter;

or,

relocalization on the target image relative to a keyframe in a keyframe database, to obtain the second pose parameter, the keyframe being an image that has successful relocalization and has a relocalization result relative to the first marker image.

12. The apparatus according to claim 11 , wherein the keyframe database comprises at least one of:

a first-order keyframe with successful first-order relocalization relative to the first marker image;

or,

an nth-order keyframe with successful nth-order relocalization relative to the (n−1) th -order keyframe in the keyframe database, n being an integer greater than 1.

13. The apparatus according to claim 10 , wherein

the front-end program is configured to calculate a current pose parameter of the current image according to the first pose parameter and the second pose parameter, comprising:

calculating a current pose parameter comprising R n and T n according to the following formula:

[

R

n

T

n

0

1

]

=

[

R

ca

s

*

T

ca

0

1

]

*

[

R

relocalize

T

relocalize

0

1

]

,

the first pose parameter comprising a first rotation matrix R ca and a first displacement vector T ca , the second pose parameter comprising a second rotation matrix R relocalize and a second displacement vector T relocalice , and s being a scale corresponding to the target image.

14. The apparatus according to claim 13 , wherein the front-end program is further configured to:

map an initial feature point in the first marker image onto the target image to obtain a projected feature point, and calculate the scale corresponding to the target image according to an average depth of the projected feature point.

15. The apparatus according to claim 10 , wherein the memory comprises a result storage space shared by the front-end program module and the back-end program module, wherein

the front-end program is configured to write the target image into the relocalization image list in the result storage space; and

the back-end program is configured to read the target image from the relocalization image list.

16. The apparatus according to claim 15 , wherein

the front-end program is configured to: detect whether a result status bit in the result storage space is a ready state; and

write the target image into the relocalization image list upon detecting that the result status bit is the ready state.

17. The apparatus according to claim 15 , wherein

the back-end program module is configured to write the second pose parameter into a result storage location in the result storage space; and

the front-end program is configured to read the second pose parameter from the result storage location.

18. The apparatus according to claim 15 , wherein

the front-end program is configured to write the target image into an expand-map list in the result storage space; and

the back-end program is configured to: read the target image from the expand-map list upon determining that the target image has successful relocalization and satisfies a preset condition, and add the target image to the keyframe database.

19. A non-transitory computer-readable storage medium, storing at least one instruction, the at least one instruction being loaded and executed by a processor to perform:

obtaining, by a front-end program run on a device, a target image acquired after an i th marker image in the plurality of marker images, i being a positive integer, the front-end program responsible for sequentially performing the camera pose tracking process on a plurality of marker images;

determining, by the front-end program, the target image as an (i+1) th marker image upon determining that the target image satisfies a relocalization condition;

transmitting, by the front-end program, the target image to a back-end program run on the device;

performing obtaining, by the front-end program, feature point tracking on a current image acquired after the target image relative to the target image to obtain a first pose parameter, the first pose parameter representing a camera pose change from the target image to the current image;

performing, by the back-end program, relocalization on the target image to obtain a second pose parameter, and transmitting the second pose parameter to the front-end program, the second pose parameter representing the camera pose change from the first marker image to the target image; and

calculating, by the front-end program, a current pose parameter of the current image according to the first pose parameter and the second pose parameter.

20. The storage medium according to claim 19 , wherein the performing, by the back-end program, relocalization on the target image to obtain a second pose parameter comprises at least one of:

performing, by the back-end program, relocalization on the target image relative to the first marker image, to obtain the second pose parameter; or,

performing, by the back-end program, relocalization on the target image relative to a keyframe in a keyframe database, to obtain the second pose parameter, the keyframe being an image that has a successful relocalization result relative to the first marker image.