Lightweight autonomous system evaluation platform
An example method includes (a) generating a request to obtain log data descriptive of operation of an autonomous vehicle control system in a subject scenario, the log data indexed by an indexing parameter; (b) submitting the request to a log repository service, the log repository service configured to receive the request and serve, responsive to the request, log metadata; (c) loading, based on the log metadata and an index value of the indexing parameter, a log data sketch associated with the index value; and (d) loading, based on the log metadata and the index value, and responsive to an inspection indicator, detailed log data associated with the index value.
1 . A computer-implemented method for autonomous vehicle control system evaluation, comprising:
(a) generating a request to obtain log data descriptive of operation of an autonomous vehicle control system in a subject scenario, the log data indexed by an indexing parameter;
(b) submitting the request to a log repository service, the log repository service configured to receive the request and serve, responsive to the request, log metadata;
(c) loading, based on the log metadata and an index value of the indexing parameter, a log data sketch associated with the index value;
(d) generating, using a machine-learned model, a data tier corresponding to the log data based on the log data sketch, wherein:
the data tier is associated with a priority of the log data based on a predicted indication for further evaluation, and
the data tier corresponds to at least a first latency budget and a second latency budget wherein the first latency budget and the second latency budget are associated with loading the log data at different speeds according to the priority; and
(e) loading, based on the data tier, the log metadata, and the index value, and responsive to an inspection indicator, detailed log data associated with the index value.
2 . The computer-implemented method of claim 1 , wherein:
the log data sketch comprises data at a first fidelity with respect to the log data; and
the detailed log data comprises data at a second fidelity higher than the first fidelity.
3 . The computer-implemented method of claim 1 , wherein loading the log data sketch comprises rendering the log data sketch on a display interface.
4 . The computer-implemented method of claim 1 , wherein a user input sequence comprises a plurality of index values of the indexing parameter, and wherein the inspection indicator corresponds to a selection of a particular index value of the plurality of index values.
5 . The computer-implemented method of claim 1 , wherein a sketch compression value associated with the log data sketch is adjusted based on a communication speed between a log data repository and an evaluation interface.
6 . The computer-implemented method of claim 1 comprising:
(f) modifying, from a default value, one or more parameters of the log metadata to customize contents of the log data sketch.
7 . The computer-implemented method of claim 1 , wherein the detailed log data is dynamically loaded or output on-demand.
8 . The computer-implemented method of claim 1 , wherein the inspection indicator corresponds to a user input.
9 . The computer-implemented method of claim 8 , wherein a user input sequence corresponds to a scrubbing input, and wherein a selection of a particular index value corresponds to a rate change in the scrubbing input associated with the particular index value.
10 . The computer-implemented method of claim 1 , wherein the indexing parameter is time.
11 . The computer-implemented method of claim 1 , wherein:
the log data sketch comprises data at a first resolution; and
the detailed log data comprises data at a second resolution higher than the first resolution.
12 . The computer-implemented method of claim 1 , wherein the log data sketch comprises a downsampled representation of the log data.
13 . The computer-implemented method of claim 1 , wherein the log data sketch comprises a reduced range of log data values from the log data.
14 . The computer-implemented method of claim 1 , wherein the log data sketch is loaded by an evaluation interface that runs in a browser, and wherein the log metadata is retrieved upon loading of an initial set of markup instructions for the evaluation interface.
15 . A computing system comprising:
one or more processors; and
one or more non-transitory computer-readable media storing instructions executable by the one or more processors to cause the computing system to perform operations, the operations comprising:
(a) generating a request to obtain log data descriptive of operation of an autonomous vehicle control system in a subject scenario, the log data indexed by an indexing parameter;
(b) submitting the request to a log repository service, the log repository service configured to receive the request and serve, responsive to the request, log metadata;
(c) loading, based on the log metadata and an index value of the indexing parameter, a log data sketch associated with the index value;
(d) generating, using a machine-learned model, a data tier corresponding to the log data based on the log data sketch, wherein:
the data tier is associated with a priority of the log data based on a predicted indication for further evaluation, and
the data tier corresponds to at least a first latency budget and a second latency budget wherein the first latency budget and the second latency budget are associated with loading the log data at different speeds according to the priority; and
(e) loading, based on the data tier, the log metadata, and the index value, and responsive to an inspection indicator, detailed log data associated with the index value.
16 . The computing system of claim 15 , wherein:
the log data sketch comprises data at a first fidelity with respect to the log data; and
the detailed log data comprises data at a second fidelity higher than the first fidelity.
17 . The computing system of claim 15 , wherein loading the log data sketch comprises rendering the log data sketch on a display interface.
18 . The computing system of claim 15 , wherein a user input sequence comprises a plurality of index values of the indexing parameter, and wherein the inspection indicator corresponds to a selection of a particular index value of the plurality of index values.
19 . The computing system of claim 15 , the operations comprising:
(D) modifying, from a default value, one or more parameters of the log metadata to customize contents of the log data sketch.
20 . One or more non-transitory computer-readable media storing instructions executable by one or more processors to cause a computing system to perform operations, the operations comprising:
(a) generating a request to obtain log data descriptive of operation of an autonomous vehicle control system in a subject scenario, the log data indexed by an indexing parameter;
(b) submitting the request to a log repository service, the log repository service configured to receive the request and serve, responsive to the request, log metadata;
(c) loading, based on the log metadata and an index value of the indexing parameter, a log data sketch associated with the index value;
(d) generating, using a machine-learned model, a data tier corresponding to the log data based on the log data sketch, wherein:
the data tier associated with a priority of the log data based on a predicted indication for further evaluation, and
the data tier corresponds to at least a first latency budget and a second latency budget wherein the first latency budget and the second latency budget are associated with loading the log data at different speeds according to the priority; and
(e) loading, based on the data tier, the log metadata, and the index value, and responsive to an inspection indicator, detailed log data associated with the index value.