Systems and methods for evaluating perception systems for autonomous vehicles using quality temporal logic

1. A framework for evaluating the quality of a vehicle perception system, the framework comprising:

a processor operable for executing instructions comprising:

extracting an object index and a set of data attributes from a frame of an object detection data stream, wherein the object index and the set of data attributes are associated with an object detected by a vehicle perception system;

receiving a set of quality predicates descriptive of expected behavior of the vehicle perception system over a plurality of frames, wherein each of the set of quality predicates is associated with a frame, wherein each of the set of quality predicates is associated with a scoring function of a plurality of scoring functions and wherein the set of quality predicates descriptive of expected behavior of the vehicle perception system over the plurality of frames are defined in terms of a Timed Quality Temporal Logic syntax;

evaluating each of the plurality of scoring functions based on observed behavior of the vehicle perception system over the plurality of frames, wherein each of the plurality of scoring functions extracts information associated with the object using the set of data attributes and compares the information with a constant quality value and wherein a result of the scoring function is descriptive of observed behavior of the vehicle perception system with respect to the associated quality predicate; and

identifying one or more quality predicates of the plurality of quality predicates that the vehicle perception system does not satisfy based on the evaluation of the plurality of scoring functions indicative of observed behavior of the vehicle perception system with respect to expected behavior of the vehicle perception system over the plurality of frames.

2. The framework of claim 1 , wherein each of the plurality of scoring functions comprises an application-specific quality function, wherein the application-specific quality function returns a quality value about the set of data attributes indicative of observed behavior of the vehicle perception system over the plurality of frames with respect to expected behavior of the vehicle perception system over the plurality of frames as defined by the associated quality predicate.

3. The framework of claim 2 , wherein the quality value returned from the application-specific quality function is used by the scoring function to determine the quality value associated with the quality predicate of the set of quality predicates, wherein the quality value is indicative of how the observed behavior of the vehicle perception system adheres to expected behavior of the vehicle perception system over the plurality of frames.

4. The framework of claim 1 , wherein the object detection data stream is provided by a perception algorithm of the vehicle perception system.

5. The framework of claim 4 , wherein a set-of-objects function is operable to retrieve the object index from a frame of the object detection stream and wherein the set-of-objects function is specific to the perception algorithm.

6. The framework of claim 4 , wherein a retrieval function retrieves the set of data attributes from a frame of the object detection stream and wherein the retrieval function is specific to the perception algorithm.

7. The framework of claim 4 , wherein the object detection stream is a tuple comprised of a plurality of data types.

8. The framework of claim 7 , wherein the tuple comprises an object index and a plurality of data attributes, wherein the plurality of data attributes comprises a probability associated with the object, a class associated with the object, and a set of bounding box coordinates associated with the object.

9. The framework of claim 2 , wherein the application-specific quality function may be defined by a user for a particular application.

10. The framework of claim 1 , wherein the scoring function returns a Boolean result when comparing data attributes from sets without scalar metrics.