Snapshots for evaluating component operation in autonomous vehicles
The technology involves evaluating components of an autonomous vehicle that can be meaningfully evaluated over a single operational iteration. One or more snapshots of single iteration scenarios can be tested quickly and efficiently, either on vehicle or via a back-end system. Each snapshot corresponds to a particular point in time when a given component runs. Each snapshot contains a serialized set of inputs necessary to evaluate the particular component. These inputs comprise the minimal amount of information needed to accurately and faithfully recreate what the component did or does. Each snapshot is triggered at the particular point in time based on one or more criteria associated with either a driving scenario or a component of the vehicle during autonomous driving. A serialized snapshot may be retrieved from storage and deserialized, so that the system may evaluate the state of the component at the particular point in time.
1 . A method for evaluating a component of a vehicle configured to operate in an autonomous driving mode, the method comprising:
triggering generation of a snapshot at a particular point in time based on a criterion associated with either a driving scenario or a component of the vehicle while operating in the autonomous driving mode, the generated snapshot being associated with the component and capturing a set of inputs necessary to run the component in order to identify either a state of the component or an output of the component at the particular point in time;
serializing, by one or more processors of a computing device, at least a portion of a set of state information corresponding to the generated snapshot, the set of state information being a minimal amount of information needed to recreate an action or a state of the component at the particular point in time;
storing, by one or more processors, the generated snapshot with the serialized portion of the set of state information in memory;
performing, by the one or more processors, a deserializing operation on the serialized portion of the set of state information; and
providing the generated snapshot and the deserialized portion of the set of state information for evaluation of the state of the component or the output of the component at the particular point in time.
2 . The method of claim 1 , wherein the generated snapshot is stored in the memory in a format that is backward and forward compatible.
3 . The method of claim 1 , wherein prior to serializing, the snapshot is published via an application programming interface of the vehicle.
4 . The method of claim 1 , wherein the generation of the snapshot is triggered during real-time driving by the vehicle in the autonomous driving mode.
5 . The method of claim 1 , wherein the generation of the snapshot is triggered during a simulation associated with operation of the vehicle in the autonomous driving mode.
6 . The method of claim 1 , wherein the criterion is associated with a flag, a timer, an explicit message request received from an on-board module of the vehicle, or an explicit message request received from a remote service.
7 . The method of claim 1 , wherein providing the generated snapshot comprises publishing a message via an internal communication bus of the vehicle.
8 . The method of claim 1 , wherein the component is either a planner module, a behavior prediction module or a perception module of the vehicle.
9 . The method of claim 1 , further comprising:
receiving a request for retrieval of the generated snapshot;
wherein the deserializing operation is performed in response to receiving the request for retrieval.
10 . A method for evaluating a component of a vehicle configured to operate in an autonomous driving mode, the method comprising:
receiving, by one or more processors, a request to invoke a snapshot, the snapshot, at generation thereof, capturing a set of inputs necessary to run the component of the vehicle in order to identify either a state of the component or an output of the component at a particular point in time;
retrieving from a database, by the one or more processors, the snapshot including a set of state information, the set of state information being a minimal amount of information needed to recreate an action or the state of the component at the particular point in time, and wherein at least a portion of the set of state information is stored in a serialized format;
performing, by the one or more processors, a deserializing operation on the portion of the set of state information; and
identifying, by the one or more processors based at least on the snapshot and the deserialized portion of the set of state information, either the state of the component or the output of the component at a particular point in time.
11 . The method of claim 10 , wherein identifying either the state of the component or the output of the component includes:
initializing internal structures of the component so that they are equivalent to how the internal structures were at the generation of the snapshot; and
invoking the component by passing in the initialized internal structures.
12 . The method of claim 10 , further comprising evaluating the output of the component to determine whether the component found a deterministic solution to an input condition.
13 . The method of claim 10 , ifurther comprising:
retrieving a pre-serialized version of the portion of the set of state information; and
comparing the deserialized portion of the set of state information to the pre-serialized version of the portion of the set of state information.
14 . The method of claim 13 , further comprising, responsive to the deserialized portion of the set of state information not conforming to the pre-serialized version of the portion of the set of state information, performing a debug operation to flag any discrepancy between the deserialized portion of the set of state information and the pre-serialized version of the portion of the set of state information.
15 . The method of claim 10 , further comprising prior to receiving the request to invoke the snapshot:
altering one or more items in the set of state information in order to result in a change to either the state of the component or the output of the component at the particular point in time.
16 . The method of claim 15 , further comprising:
repeating the altering to create a set of snapshots, each snapshot in the set having a different set of altered items in the set of state information.
17 . The method of claim 16 , further comprising:
defining one or more metrics for the component based on evaluation of the set of snapshots.
18 . The method of claim 15 , wherein altering the one or more items in the set of state information alters at least one of a behavior prediction likelihood or a trajectory path associated with an operation by the component.
19 . A processing system, comprising:
memory configured to store a snapshot, the snapshot capturing a set of inputs necessary to run a component of a vehicle in order to identify either a state of the component or an output of the component at a particular point in time; and
one or more processors operatively coupled to the memory, the one or more processors being configured to:
trigger generation of the snapshot at the particular point in time based on a criterion associated with either a driving scenario or a component of the vehicle while operating in an autonomous driving mode;
serialize, for storage of the snapshot in the memory, at least a portion of a set of state information corresponding to the snapshot, the set of state information being a minimal amount of information needed to recreate an action or a state of the component at the particular point in time;
subsequent to serialization of the portion of the set of state information, store the snapshot with the set of state information in the memory;
subsequent to storage of the snapshot and the set of state information in the memory, perform a deserializing operation on the portion of the set of state information; and
provide the snapshot with the deserialized portion of the set of state information for evaluation of the state of the component or the output of the component at the particular point in time.
20 . The processing system of claim 19 , wherein the one or more processors are further configured to:
trigger the generation of the snapshot during real-time driving by the vehicle in the autonomous driving mode; and
provide the snapshot by being configured to publish a message via an internal communication bus of the vehicle.
21 . The method of claim 1 , wherein serializing the at least the portion of the set of state information is performed in association with storing the generated snapshot and the serialized portion of the set of state information in the memory.
22 . The method of claim 1 , wherein serializing the at least the portion of the set of state information includes serializing the at least the portion of the set of state information into a protocol buffer prior to storing the generated snapshot with the serialized portion of the set of state information in the memory.