Granted Patent B2

US 12,276,507 · App. 17/645,449 · Granted Apr 15, 2025

Inventors: Erli Meng (Beijing, CN); Luting Wang (Beijing, CN)

Assignees: BEIJING XIAOMI MOBILE SOFTWARE CO., LTD.; BEIJING XIAOMI PINECONE ELECTRONICS CO., LTD.

G01C21/206G06T1/0014G06T7/73G06V10/426G06V10/7747G06V10/806G06V20/36G06T2207/20072

Abstract

Claims (91)

1. An indoor navigation method, applied to a navigation equipment, wherein the indoor navigation method comprises:

receiving an instruction for navigation, and collecting an environment image;

extracting an instruction room feature and an instruction object feature carried in the instruction, and determining a visual room feature, a visual object feature, and a view angle feature based on the environment image, wherein the instruction room feature is configured to indicate room information obtained from the instruction for navigation, the instruction object feature is configured to indicate object information obtained from the instruction for navigation, the visual room feature is configured to indicate room information obtained from the environment image, the visual object feature is configured to indicate object information obtained from the environment image, and the view angle feature is configured to reflect information carried in an view angle of the environment image;

fusing the instruction object feature and the visual object feature with a first knowledge graph representing an indoor object association relationship to obtain an object feature, and determining a room feature based on the visual room feature and the instruction room feature; and

determining a navigation decision based on the view angle feature, the room feature, and the object feature;

wherein fusing the instruction object feature and the visual object feature with the first knowledge graph representing the indoor object association relationship to obtain the object feature comprises:

extracting an object entity carried in the environment image based on the visual object feature;

constructing a second knowledge graph based on the object entity and the first knowledge graph representing the indoor object association relationship, wherein the second knowledge graph is configured to represent an association relationship between the object entity and a first object entity in the first knowledge graph that has an association relationship with the object entity;

performing multi-step graph convolutional reasoning on the first knowledge graph and the second knowledge graph respectively so as to obtain first knowledge graph reasoning information and second knowledge graph reasoning information;

fusing the first knowledge graph reasoning information with the second knowledge graph reasoning information, and updating the first knowledge graph by using the fused knowledge graph reasoning information;

performing a first feature fusing and reinforcing operation on the instruction object feature based on the second knowledge graph to obtain an enhanced instruction object feature; and

performing a second feature fusing and reinforcing operation on the updated first knowledge graph and the enhanced instruction object feature to obtain the object feature; and

wherein determining the room feature based on the visual room feature and the instruction room feature comprises:

determining a visual room category carried in each of optional view angles based on the visual room feature, and determining an instruction room category carried in each of the optional view angles based on the instruction room feature;

determining a room confidence level of each of the optional view angles based on the visual room category, the instruction room category, and a preset room correlation matrix; and

determining the room feature based on the room confidence level of each of the optional view angles.

2. The indoor navigation method according to claim 1 , wherein determining the navigation decision based on the view angle feature, the room feature, and the object feature comprises:

determining a total view angle feature of the environment image based on the view angle feature;

splicing the total view angle feature, the object feature, the instruction room feature, and the instruction object feature to obtain a scenario memory token and obtain current navigation progress information based on the scenario memory token, and splicing the room feature and the view angle feature to obtain an optional view angle feature; and

performing a third feature fusing and reinforcing operation on the optional view angle feature and the current navigation progress information to obtain a navigation decision for a next navigation progress.

3. The indoor navigation method according to claim 2 , wherein determining the total view angle feature of the environment image based on the view angle feature comprises:

determining a previous navigation progress information; and

performing a fourth feature fusing and reinforcing operation on the view angle feature and the previous navigation progress information to obtain the total view angle feature of the environment image.

4. The indoor navigation method according to claim 1 , wherein the indoor navigation method further comprises:

determining a penalty coefficient based on a deviation degree between a view angle in the navigation decision and an optimal view angle, wherein the optimal view angle is an optional view angle that is closest to a navigation end point among all the optional view angles; and

changing the view angle in the navigation decision based on the penalty coefficient.

5. The indoor navigation method according to claim 4 , wherein determining the navigation decision comprises:

determining the navigation decision based on a vision-language navigation model;

obtaining the vision-language navigation model by training by determining a total loss function based on an imitation learning loss function, a room category prediction loss function, and a direction perception loss function, the imitation learning loss function is configured to represent a deviation degree between the optional view angle and the optimal view angle, the room category prediction loss function is configured to represent a deviation degree between a room category corresponding to the optional view angle and a room category in the navigation decision, and the direction perception loss function is configured to represent a deviation degree between the view angle in the navigation decision and the optimal view angle; and

training the vision-language navigation model based on the total loss function.

6. The indoor navigation method according to claim 1 , wherein the indoor navigation method further comprises:

determining a value of logit of each of the optional view angles, and determining a backtracking distance between each of the optional view angles and a current position; and

modifying the value of logit of each of the optional view angles based on the backtracking distance, and changing the view angle in the navigation decision based on the modified values of logit.

7. An indoor navigation equipment, comprising:

one or more processors; and

one or more memories configured to store instructions executable by the processor;

wherein the processor is configured to:

receive an instruction for navigation, and collect an environment image;

extract an instruction room feature and an instruction object feature carried in the instruction, and determine a visual room feature, a visual object feature, and a view angle feature based on the environment image, wherein the instruction room feature is configured to indicate room information obtained from the instruction for navigation, the instruction object feature is configured to indicate object information obtained from the instruction for navigation, the visual room feature is configured to indicate room information obtained from the environment image, the visual object feature is configured to indicate object information obtained from the environment image, and the view angle feature is configured to reflect information carried in an view angle of the environment image;

fuse the instruction object feature and the visual object feature with a first knowledge graph representing an indoor object association relationship to obtain an object feature, and determine a room feature based on the visual room feature and the instruction room feature; and

determine a navigation decision based on the view angle feature, the room feature, and the object feature;

wherein the processor is further configured to:

extract an object entity carried in the environment image based on the visual object feature;

construct a second knowledge graph based on the object entity and the first knowledge graph representing the indoor object association relationship, wherein the second knowledge graph is configured to represent an association relationship between the object entity and a first object entity in the first knowledge graph that has an association relationship with the object entity;

perform multi-step graph convolutional reasoning on the first knowledge graph and the second knowledge graph respectively so as to obtain first knowledge graph reasoning information and second knowledge graph reasoning information;

fuse the first knowledge graph reasoning information with the second knowledge graph reasoning information, and update the first knowledge graph by using the fused knowledge graph reasoning information;

perform a first feature fusing and reinforcing operation on the instruction object feature based on the second knowledge graph to obtain an enhanced instruction object feature; and

perform a second feature fusing and reinforcing operation on the updated first knowledge graph and the enhanced instruction object feature to obtain the object feature; and

wherein the processor is further configured to:

determine a visual room category carried in each of optional view angles based on the visual room feature, and determine an instruction room category carried in each of the optional view angles based on the instruction room feature;

determine a room confidence level of each of the optional view angles based on the visual room category, the instruction room category, and a preset room correlation matrix; and

determine the room feature based on the room confidence level of each of the optional view angles.

8. The indoor navigation equipment according to claim 7 , wherein the processor is further configured to:

determine a total view angle feature of the environment image based on the view angle feature;

splice the total view angle feature, the object feature, the instruction room feature, and the instruction object feature to obtain a scenario memory token and obtain current navigation progress information based on the scenario memory token, and splice the room feature and the view angle feature to obtain an optional view angle feature; and

perform a third feature fusing and reinforcing operation on the optional view angle feature and the current navigation progress information to obtain a navigation decision for a next navigation progress.

9. The indoor navigation equipment according to claim 8 , wherein the processor is further configured to:

determine a previous navigation progress information; and

perform a fourth feature fusing and reinforcing operation on the view angle feature and the previous navigation progress information to obtain the total view angle feature of the environment image.

10. The indoor navigation equipment according to claim 7 , wherein the processor is further configured to:

determine a penalty coefficient based on a deviation degree between a view angle in the navigation decision and an optimal view angle, wherein the optimal view angle is an optional view angle that is closest to a navigation end point among all the optional view angles; and

change the view angle in the navigation decision based on the penalty coefficient.

11. The indoor navigation equipment according to claim 10 , wherein the processor is further configured to:

determine the navigation decision based on a vision-language navigation model;

obtain the vision-language navigation model by training by determining a total loss function based on an imitation learning loss function, a room category prediction loss function, and a direction perception loss function, the imitation learning loss function is configured to represent a deviation degree between the optional view angle and the optimal view angle, the room category prediction loss function is configured to represent a deviation degree between a room category corresponding to the optional view angle and a room category in the navigation decision, and the direction perception loss function is configured to represent a deviation degree between the view angle in the navigation decision and the optimal view angle; and

train the vision-language navigation model based on the total loss function.

12. The indoor navigation equipment according to claim 7 , wherein the processor is further configured to:

determine a value of logit of each of the optional view angles, and determine a backtracking distance between each of the optional view angles and a current position; and

modify the value of logit of each of the optional view angles based on the backtracking distance, and change the view angle in the navigation decision based on the modified values of logit.

13. A non-transitory computer-readable storage medium, wherein when an instruction in the storage medium is executed by a processor of a mobile terminal, the mobile terminal is capable of executing an indoor navigation method, and the method comprises:

receiving an instruction for navigation, and collecting an environment image;

extracting an instruction room feature and an instruction object feature carried in the instruction, and determining a visual room feature, a visual object feature, and a view angle feature based on the environment image, wherein the instruction room feature is configured to indicate room information obtained from the instruction for navigation, the instruction object feature is configured to indicate object information obtained from the instruction for navigation, the visual room feature is configured to indicate room information obtained from the environment image, the visual object feature is configured to indicate object information obtained from the environment image, and the view angle feature is configured to reflect information carried in an view angle of the environment image;

fusing the instruction object feature and the visual object feature with a first knowledge graph representing an indoor object association relationship to obtain an object feature, and determining a room feature based on the visual room feature and the instruction room feature; and

determining a navigation decision based on the view angle feature, the room feature, and the object feature;

wherein fusing the instruction object feature and the visual object feature with a first knowledge graph representing the indoor object association relationship to obtain the object feature comprises:

extracting an object entity carried in the environment image based on the visual object feature;

constructing a second knowledge graph based on the object entity and the first knowledge graph representing the indoor object association relationship, wherein the second knowledge graph is configured to represent an association relationship between the object entity and a first object entity in the first knowledge graph that has an association relationship with the object entity;

performing multi-step graph convolutional reasoning on the first knowledge graph and the second knowledge graph respectively so as to obtain first knowledge graph reasoning information and second knowledge graph reasoning information;

fusing the first knowledge graph reasoning information with the second knowledge graph reasoning information, and updating the first knowledge graph by using the fused knowledge graph reasoning information;

performing a first feature fusing and reinforcing operation on the instruction object feature based on the second knowledge graph to obtain an enhanced instruction object feature; and

performing a second feature fusing and reinforcing operation on the updated first knowledge graph and the enhanced instruction object feature to obtain the object feature; and

wherein determining the room feature based on the visual room feature and the instruction room feature comprises:

determining a visual room category carried in each of optional view angles based on the visual room feature, and determining an instruction room category carried in each of the optional view angles based on the instruction room feature;

determining a room confidence level of each of the optional view angles based on the visual room category, the instruction room category, and a preset room correlation matrix; and

determining the room feature based on the room confidence level of each of the optional view angles.

14. The non-transitory computer-readable storage medium according to claim 13 , wherein the indoor navigation method further comprises:

determining a penalty coefficient based on a deviation degree between a view angle in the navigation decision and an optimal view angle, wherein the optimal view angle is an optional view angle that is closest to a navigation end point among all the optional view angles; and

changing the view angle in the navigation decision based on the penalty coefficient.

15. The storage medium according to claim 13 , wherein the indoor navigation method further comprises:

determining a value of logit of each of the optional view angles, and determining a backtracking distance between each of the optional view angles and a current position; and

modifying the value of logit of each of the optional view angles based on the backtracking distance, and changing the view angle in the navigation decision based on the modified values of logit.

Assignments (1)

Priority Claims (1)

Continuity (1)

References Cited (41)

Cited By (1)

US 12,718,548