Displaying updated vehicle behavior in real time visualizations
Aspects of the technology employs snapshots, which are serialized sets of inputs for evaluating a components of a vehicle's autonomous driving system. Snapshots can be performed very quickly with minimal compile latency and runtime, allowing rapid evaluation of individual autonomous vehicle elements in isolation. Parameter variation and changes to initial conditions associated with a given component may be applied via a robust user interface (UI), enabling a user to perform permutations to a component or scenario and receive immediate feedback, e.g., graphically presented in a three-dimensional (3D) scene to understand how the vehicle is operating. The UI may include various control elements to enable real-time adjustment to the parameters or conditions of a snapshot. This can include one or more mutations of a snapshot to evaluate different situations or conditions, which can further aid in understanding how the vehicle would function in an autonomous driving mode.
1 . A method for evaluating a component of a vehicle configured to operate in an autonomous driving mode, the method comprising:
retrieving from a database, by one or more processors,—a snapshot associated with the component and including a set of state information, the set of state information comprising a minimal amount of information needed to recreate an action or a state of the component at a particular point in time, and wherein at least a portion of the set of state information is stored in a serialized format;
performing, by the one or more processors, a deserializing operation on the portion of the set of-state information to yield a deserialized portion of the set of state information;
receiving, by the one or more processors, user input indicating a request to visualize the snapshot, the request including mutating a feature of the snapshot, the mutating including modifying a property of an object in an environment of the vehicle;
in response to the request, identifying, by the one or more processors based at least on the deserialized portion of the set of state information, either the state of the component or an output of the component at the particular point in time, the identifying including initializing internal structures of the component in accordance with mutating the feature; and
in according with the identifying, generating, by the one or more processors for display on a display device, a visualization of the snapshot, the visualization indicating the state of the component or the output of the component.
2 . The method of claim 1 , wherein identifying either the state of the component or the output of the component includes initializing internal structures of the component so that they are equivalent to how the internal structures were at generation of the snapshot.
3 . The method of claim 2 , wherein the internal structures are initialized based on one or more cached resources maintained separately from the snapshot.
4 . The method of claim 3 , wherein the one or more cached resources include at least one of a roadgraph, a machine learning model, or an output from another component of the vehicle.
5 . The method of claim 1 , wherein mutating the feature includes either adding or removing another object in the environment of the vehicle.
6 . The method of claim 1 , wherein mutating the feature further includes modulating a tuning parameter associated with operating the vehicle in the autonomous driving mode.
7 . The method of claim 1 , wherein mutating the feature further includes modifying one or more aspects of a behavior prediction for an agent, the one or more aspects including at least one of state positions of the agent or a behavior prediction likelihood.
8 . The method of claim 7 , wherein the behavior prediction is conditional to a future action.
9 . The method of claim 1 , wherein mutating the feature further includes selecting a target reference path.
10 . The method of claim 1 , wherein identifying either the state of the component or the output of the component includes analyzing any flags specified at runtime for the snapshot.
11 . The method of claim 1 , further comprising, in response to the user input, performing a toggle operation between two or more versions of the snapshot.
12 . The method of claim 1 , further comprising storing the visualization of the snapshot in the database.
13 . The method of claim 1 , further comprising, in response to the user input, generating a set of mutated snapshots, each mutated snapshot in the set having a different group of features.
14 . A processing system, comprising:
memory configured to store a snapshot, the snapshot being associated with a component of a vehicle and capturing a set of inputs necessary to run the component in order to identify either a state of the component or an output of the component at a particular point in time, the vehicle being configured to operate in an autonomous driving mode; and
one or more processors operatively coupled to the memory, the one or more processors being configured to:
retrieve the snapshot from the memory, the snapshot including a set of state information, the set of state information comprising a minimal amount of information needed to recreate an action or the state of the component at the particular point in time, and wherein at least a portion of the set of state information is stored in the memory in a serialized format;
perform a deserializing operation on the portion of the set of state information to yield a deserialized portion of the set of state information;
receive user input indicating a request to visualize the snapshot, the request including mutating a feature of the snapshot, the mutating including modulating a tuning parameter associated with operating the vehicle in the autonomous driving mode;
in response to the request, identify, based at least on the deserialized portion of the set of state information, either the state of the component or the output of the component at the particular point in time, the identifying including initializing internal structures of the component in accordance with mutating the feature; and
in according with the identification, generate, for display on a display device, a visualization of the snapshot, the visualization indicating the state of the component or the output of the component.
15 . The processing system of claim 14 , wherein the one or more processors are further configured to cache information about the snapshot in order to either reduce an amount of time to evaluate modified versions of the snapshot or generate visualizations of the modified versions of the snapshot.
16 . A method for evaluating a component of a vehicle configured to operate in an autonomous driving mode, the method comprising:
retrieving from a database, by one or more processors, —a snapshot associated with the component and including a set of state information, the set of state information comprising a minimal amount of information needed to recreate an action or a state of the component at a particular point in time, and wherein at least a portion of the set of state information is stored in a serialized format;
performing, by the one or more processors, a deserializing operation on the portion of the set of-state information to yield a deserialized portion of the set of state information;
receiving, by the one or more processors, user input indicating a request to visualize the snapshot, the request including mutating a feature of the snapshot, the mutating including selecting a target reference path;
in response to the request, identifying, by the one or more processors based at least on the deserialized portion of the set of state information, either the state of the component or an output of the component at the particular point in time, the identifying including initializing internal structures of the component in accordance with mutating the feature; and
in according with the identifying, generating, by the one or more processors for display on a display device, a visualization of the snapshot, the visualization indicating the state of the component or the output of the component.
17 . A method for evaluating a component of a vehicle configured to operate in an autonomous driving mode, the method comprising:
retrieving from a database, by one or more processors,—a snapshot associated with the component and including a set of state information, the set of state information comprising a minimal amount of information needed to recreate an action or a state of the component at a particular point in time, and wherein at least a portion of the set of state information is stored in a serialized format;
performing, by the one or more processors, a deserializing operation on the portion of the set of-state information to yield a deserialized portion of the set of state information;
receiving, by the one or more processors, user input indicating a request to visualize the snapshot, the request including at least one of selection of the snapshot, mutating a feature of the snapshot, or toggling between two or more versions of the snapshot;
in response to the request, identifying, by the one or more processors based at least on the deserialized portion of the set of state information, either the state of the component or an output of the component at the particular point in time;
in according with the identifying, generating, by the one or more processors for display on a display device, a visualization of the snapshot, the visualization indicating the state of the component or the output of the component; and
in response to the user input, generating a set of mutated snapshots, each mutated snapshot in the set having a different group of features.