Systems and methods for multi-modal visual reasoning using knowledge graphs maintained over a period of time

1 . A system for processing multi-modal data representing an environment during automated operation of a vehicle, the system comprising:

one or more processors configured to:

obtain first sensor data associated with a vehicle operating in an environment at a first point in time;

generate a knowledge graph based on the first sensor data, the knowledge graph comprising (1) a plurality of nodes representing a plurality of objects and one or more agents in the environment at the first point in time, and (2) edges between respective pairs of the plurality of nodes representing relationships between objects or agents of the respective pairs of the plurality of nodes in the environment at the first point in time;

in response to obtaining second sensor data associated with the vehicle at a second point in time, determine one or more changes to the relationships involving the objects and the agents;

update the knowledge graph based on the one or more changes to the relationships by incrementally modifying the edges and the plurality of nodes based on the determined one or more changes to the relationships involving the objects and the agents; and

in response to updating the knowledge graph, provide a control signal to the vehicle to cause operation of a vehicle based on the knowledge graph based on determining the modification to the edges between the respective pairs of the plurality of nodes of the updated knowledge graph cause the updated knowledge graph to define a predefined relationship pattern indicating an unsafe condition is present in the environment.

2 . The system of claim 1 , wherein the one or more processors are further configured to:

in response to obtaining the first sensor data, determine a set of features associated with the environment based on the first sensor data, the set of features comprising the one or more objects and the one or more agents; and

generate at least one scene graph representing the environment based on the set of features,

wherein the one or more processors configured to generate the knowledge graph are configured to:

generate the knowledge graph based on the at least one scene graph.

3 . The system of claim 2 , wherein the first sensor data comprises a first portion generated during operation of a first sensor of the vehicle and a second portion generated during operation of a second sensor,

wherein the one or more processors configured to generate the at least one scene graph are configured to:

generate a first scene graph for the first portion of the first sensor data and a second scene graph for the second portion of the first sensor data.

4 . The system of claim 3 , wherein the one or more processors are further configured to:

determine a correspondence between the first portion and the second portion of the first sensor data; and

aggregate attributes from the first scene graph and the second scene graph based on the correspondence to determine a global scene graph,

wherein the one or more processors configured to generate the knowledge graph are configured to:

determine a composite representation of the environment based on attributes from the first scene graph and the second scene graph.

5 . The system of claim 1 , wherein the one or more processors are further configured to:

track movement of the one or more objects and the one or more agents in the environment based on the one or more changes to the relationships;

in response to tracking the movement of the one or more objects and the one or more agents, determining that operation of the vehicle does not satisfy one or more operational requirements at the second point in time or a third point in time; and

generate the control signal to adjust the operation of the vehicle to satisfy the one or more operational requirements at the third point in time.

6 . The system of claim 5 , wherein the one or more operational requirements comprises operating below a threshold speed when the vehicle is within a predetermined distance from the one or more objects or the one or more agents.

7 . The system of claim 5 , wherein the one or more operational requirements comprises:

operating the vehicle in accordance with a first path that is separated from a second path for objects or agents operating in the environment.

8 . The system of claim 5 , wherein the one or more processors configured to determine that operation of the vehicle does not satisfy one or more operational requirements are configured to:

determine that the vehicle is operating in accordance with a first path at least in part overlaps with one or more second paths of the one or more objects or the one or more agents.

9 . A method for processing multi-modal data representing an environment during automated operation of a vehicle, the method comprising:

obtaining first sensor data associated with a vehicle operating in an environment at a first point in time;

generating a knowledge graph based on the first sensor data, the knowledge graph comprising (1) a plurality of nodes representing a plurality of objects and one or more agents in the environment at the first point in time, and (2) edges between respective pairs of the plurality of nodes representing relationships in the environment at the first point in time;

in response to obtaining second sensor data associated with the vehicle at a second point in time, determining one or more changes to the relationships involving the objects and the agents;

updating the knowledge graph based on the one or more changes to the relationships by incrementally modifying the edges and the plurality of nodes based on the determined one or more changes to the relationships involving the objects and the agents; and

in response to updating the knowledge graph, providing a control signal to the vehicle to cause operation of a vehicle based on determining the modification to the edges between the respective pairs of the plurality of nodes of the updated knowledge graph cause the updated knowledge graph to define a predefined relationship pattern.

10 . The method of claim 9 , further comprising:

in response to obtaining the first sensor data, determining a set of features associated with the environment based on the first sensor data, the set of features comprising the one or more objects and the one or more agents; and

generating at least one scene graph representing the environment based on the set of features,

wherein generating the knowledge graph comprises:

generating the knowledge graph based on the at least one scene graph.

11 . The method of claim 10 , wherein the first sensor data comprises a first portion generated during operation of a first sensor of the vehicle and a second portion generated during operation of a second sensor,

wherein generating the at least one scene graph comprises:

generating a first scene graph for the first portion of the first sensor data and a second scene graph for the second portion of the first sensor data.

12 . The method of claim 11 , further comprising:

determining a correspondence between the first portion and the second portion of the first sensor data; and

aggregating attributes from the first scene graph and the second scene graph based on the correspondence to determine a global scene graph,

wherein generating the knowledge graph comprises:

determining a composite representation of the environment based on attributes from the first scene graph and the second scene graph.

13 . The method of claim 9 , further comprising:

tracking movement of the one or more objects and the one or more agents in the environment based on the one or more changes to the relationships;

in response to tracking the movement of the one or more objects and the one or more agents, determining that operation of the vehicle does not satisfy one or more operational requirements at the second point in time or a third point in time; and

generating the control signal to adjust the operation of the vehicle to satisfy the one or more operational requirements at the third point in time.

14 . The method of claim 13 , wherein the one or more operational requirements comprises: operating below a threshold speed when the vehicle is within a predetermined distance from the one or more objects or the one or more agents.

15 . The method of claim 13 , wherein the one or more operational requirements comprises: operating the vehicle in accordance with a first path that is separated from a second path for objects or agents operating in the environment.

16 . The method of claim 13 , wherein determining that operation of the vehicle does not satisfy one or more operational requirements comprises:

determining that the vehicle is operating in accordance with a first path at least in part overlaps with one or more second paths of the one or more objects or the one or more agents.

17 . One or more non-transitory computer-readable mediums storing instructions thereon that, when executed by one or more processors, cause the one or more processors to:

obtain first sensor data associated with a device operating in an environment at a first point in time;

generate a knowledge graph based on the first sensor data, the knowledge graph comprising (1) a plurality of nodes representing a plurality of objects and one or more agents in the environment at the first point in time, and (2) edges between respective pairs of the plurality of nodes representing relationships between objects or agents of the respective pairs of the plurality of nodes in the environment at the first point in time;

in response to obtaining second sensor data associated with the device at a second point in time, determine one or more changes to the relationships involving the objects and the agents;

update the knowledge graph based on the one or more changes to the relationships by incrementally modifying the edges and the plurality of nodes based on the determined one or more changes to the relationships involving the objects and the agents; and

in response to updating the knowledge graph, provide a control signal to the device to cause operation of a device based on determining the modification to the edges between the respective pairs of the plurality of nodes of the updated knowledge graph cause the updated knowledge graph to define a predefined relationship pattern.

18 . The one or more non-transitory computer-readable mediums of claim 17 , wherein the instructions further cause the one or more processors to:

in response to obtaining the first sensor data, determine a set of features associated with the environment based on the first sensor data, the set of features comprising the one or more objects and the one or more agents; and

generate at least one scene graph representing the environment based on the set of features,

wherein the instructions that cause the one or more processors to generate the knowledge graph cause the one or more processors to:

generate the knowledge graph based on the at least one scene graph.

19 . The one or more non-transitory computer-readable mediums of claim 18 , wherein the first sensor data comprises a first portion generated during operation of a first sensor of the device and a second portion generated during operation of a second sensor,

wherein the instructions that cause the one or more processors to generate the at least one scene graph cause the one or more processors to:

generate a first scene graph for the first portion of the first sensor data and a second scene graph for the second portion of the first sensor data.

20 . The one or more non-transitory computer-readable mediums of claim 19 , wherein the instructions further cause the one or more processors to:

determine a correspondence between the first portion and the second portion of the first sensor data; and

aggregate attributes from the first scene graph and the second scene graph based on the correspondence to determine a global scene graph,

wherein the instructions that cause the one or more processors to generate the knowledge graph cause the one or more processors to:

determine a composite representation of the environment based on attributes from the first scene graph and the second scene graph.