Performance testing for robotic systems
Herein, a “perception statistical performance model” (PSPM) for modelling a perception slice of a runtime stack for an autonomous vehicle or other robotic system may be used e.g. for safety/performance testing. A PSPM is configured to: receive a computed perception ground truth; determine from the perception ground truth, based on a set of learned parameters, a probabilistic perception uncertainty distribution, the parameters learned from a set of actual perception outputs generated using the perception slice to be modelled. The modelled perception slice includes an online error estimator, and the computer system is configured to use the PSPM to obtain a predicted online error estimate for the perception output in response to the perception ground truth. This recognizes that online perception error estimates may, themselves, be subject to error.
1. A computer system for testing and/or training a runtime stack for a robotic system, the computer system comprising:
at least one processor; and
at least one storage medium having encoded thereon executable instructions that, when executed by the at least one processor, cause the at least one processor to carry out a method comprising:
running a simulated scenario in which a simulated agent interacts with one or more external objects, wherein the runtime stack is configured to make autonomous decisions for each simulated scenario in dependence on a time series of perception outputs computed for the simulated scenario and configured to generate a series of control signals for causing the simulated agent to execute the autonomous decisions as the simulated scenario progresses;
wherein running the simulated scenario comprises computing each perception output at least in part by:
computing a perception ground truth based on a current state of the simulated scenario,
applying a perception statistical performance model (PSPM) to the perception ground truth to determine a probabilistic perception uncertainty distribution, and
sampling the perception output from the probabilistic perception uncertainty distribution;
wherein the PSPM is for modelling a perception slice of the runtime stack and is configured to determine the probabilistic perception uncertainty distribution based on a set of parameters learned from a set of actual perception outputs generated using the perception slice to be modelled; and
wherein the perception slice includes an online error estimator, and the computer system is configured to use the PSPM to obtain a predicted online error estimate for the perception output in response to the perception ground truth.
2. The computer system of claim 1 , wherein the predicted online error estimate is sampled from the probabilistic perception uncertainty distribution.
3. The computer system of claim 2 , wherein the PSPM takes the form of a function approximator that receives the perception ground truth, and outputs parameter(s) of the probabilistic perception uncertainty distribution from which the perception output and predicted online error estimate are sampled.
4. The computer system of claim 3 , wherein the PSPM has a neural network architecture.
5. The computer system of claim 1 , wherein the PSPM is applied to the perception ground truth and one or more confounders associated with the simulated scenario, wherein each confounder is a variable of the PSPM whose value characterizes a physical condition applicable to the simulated scenario and on which the probabilistic perception uncertainty distribution depends, the predicted online error estimate dependent on the confounders.
6. The computer system of claim 5 , wherein the one or more confounders comprise one or more of the following confounders, which at least partially determine the probabilistic uncertainty distribution from which the perception output is sampled:
an occlusion level for at least one of the external objects,
one or more lighting conditions,
an indication of time of day,
one or more weather conditions,
an indication of season,
a physical property of at least one of the external objects,
a sensor condition, for example a position of at least one of the external objects in a sensor field of view of the agent,
a number or density of the external objects;
a distance between two of the external objects,
a truncation level for at least one of the external objects,
a type of at least one of the external objects, or
an indication as to whether or not at least one of the external objects corresponds to any external object from an earlier time instant of the simulated scenario.
7. The computer system of claim 1 , wherein the PSPM comprises a time-dependent model such that the sampled perception output sampled at predicted online error estimate depend on at least one of: an earlier one of the perception outputs sampled at a previous time instant, or an earlier one of the perception ground truths computed for a previous time instant.
8. The computer system of claim 1 , wherein the method further comprises:
assessing behaviour of the simulated agent in each of the simulated scenarios by applying a set of predetermined rules.
9. The computer system of claim 8 , wherein:
at least some of the predetermined rules pertain to safety, and
assessing the behaviour of the simulated agent comprises assessing safety of the simulated agent's behaviour in each of the simulated scenarios.
10. The computer system of claim 1 , wherein the method further comprises recording details of each simulated scenario in a test database, wherein the details include decisions made by the runtime stack, the perception outputs on which those decisions were based, and behaviour of the simulated agent in executing those decisions.
11. The computer system of claim 1 , wherein the sampling from the probabilistic perception uncertainty distribution is non-uniform and is biased towards lower-probability perception outputs.
12. The computer system of claim 1 , wherein the method further comprises generating at least one fuzzed scenario at least in part by fuzzing at least one existing scenario.
13. The computer system of claim 1 , wherein, to model false negative detections, the probabilistic perception uncertainty distribution provides a probability of successfully detecting a visible one of the external objects, which is used to determine whether or not to provide an object detection output for that external object, an external object being visible when it is within a sensor field of view of the simulated agent in the simulated scenario, whereby detection of the visible external object is not guaranteed.
14. The computer system of claim 1 , wherein the perception ground truths are computed for the one or more external objects using ray tracing.
15. The computer system of claim 1 , wherein at least one of the external objects is a moving actor, the computer system comprising a prediction stack of the runtime stack configured to predict behaviour of the external actor based on the perception outputs, the runtime stack configured to make the autonomous decisions in dependence on the predicted behaviour.
16. A computer-implemented method of performance testing a runtime stack for a robotic system, the method comprising:
running a simulated scenario in which a simulated agent interacts with one or more external objects, wherein the runtime stack makes autonomous decisions for the simulated scenario in dependence on a time series of perception outputs computed for the simulated scenario and generates a series of control signals for causing the simulated agent to execute the autonomous decisions as the simulated scenario progresses;
wherein each perception output is computed by:
computing a perception ground truth based on a current state of the simulated scenario,
applying a perception statistical performance model (PSPM) to the perception ground truth to determine a probabilistic perception uncertainty distribution, and
sampling the perception output from the probabilistic perception uncertainty distribution;
wherein the PSPM is for modelling a perception slice of the runtime stack and determined the probabilistic perception uncertainty distribution based on a set of parameters learned from a set of actual perception outputs generated using the perception slice to be modelled; and
wherein the perception slice includes an online error estimator, and the PSPM is used to obtain a predicted online error estimate for the perception output in response to the perception ground truth.