Automatic assessment of unsupervised models via trust scoring in unsupervised edge domains
Model assessment is disclosed. When a model operates, tuples are transmitted to a central node. The central node can process the tuples received from multiple nodes to generate an efficiency score for the model. The efficiency score reflects how the inference of the model correlates to operator actions. Models whose assessment is below a threshold score may be retrained at least for certain classes.
1 . A method for assessing a quality of a performance of a model in an unsupervised manner without labelled data, the method comprising:
receiving tuples from a node comprising one or more sensors,
wherein each tuple includes an operator identifier of a human operator, an inference, a class, and an operator action done by the human operator,
wherein the operator action is captured by a portion of the one or more sensors and generated by the model based on sensor data collected by the portion of the one or more sensors,
wherein the inference is predicted by the model operating on the node based on the sensor data collected by the portion of the one or more sensors,
wherein the inference represents an indication of an event, and
wherein the class, for each of the tuples, is one of multiple classes and is a trajectory class which is determined from the sensor data;
storing the tuples in a first table;
generating a second table that associates the human operator with classes from the first table;
generating a prediction-action table that associates the inference to the operator action using a specific class from the second table, wherein the prediction-action table captures relationships between inferences and operator actions;
generating an efficiency score from the prediction-action table by comparing the inference and the operator action in each tuple;
determining an efficiency score distribution for the class;
assessing quality of the performance of the model based on the efficiency score and the efficiency score distribution; and
retraining the model when the efficiency score distribution indicates that the human operator is not reacting in accordance with the inferences.
2 . The method of claim 1 , wherein the inference is Boolean and includes two types, and
wherein the method further comprises generating a scenario efficiency for each of the two types.
3 . The method of claim 2 , further comprising weighting each of the scenario efficiencies and summing the weighted scenario efficiencies to determine the efficiency score.
4 . The method of claim 3 , wherein an efficiency score is generated for each of the classes represented in the second table.
5 . The method of claim 4 , further comprising generating the inference based on sensor data collected from sensors operating at the node.
6 . The method of claim 1 , further comprising training the model.
7 . The method of claim 1 , further comprising determining an efficiency score distribution for the model for each of the classes and assessing the quality of the performance of the model for each of the classes.
8 . The method of claim 7 , further comprising retraining the model for scenarios under a threshold efficiency score.
9 . The method of claim 8 , further comprising determining the class from sensor data generated at the node using a first model and determining the inference from a second model.
10 . The method of claim 1 , wherein the model detects cornering events,
wherein the inference determines whether the cornering event is safe or dangerous, and
wherein the operator action is correlated to the trajectory class, the inference, and the operator identifier.
11 . A non-transitory storage medium having stored therein instructions that are executable by one or more hardware processors to perform operations for assessing a quality of a performance of a model in an unsupervised manner without labelled data, the operations comprising:
receiving tuples from a node comprising one or more sensors,
wherein each tuple includes an operator identifier of a human operator, an inference, a class, and an operator action done by the human operator,
wherein the operator action is captured by a portion of the one or more sensors and generated by the model based on sensor data collected by the portion of the one or more sensors,
wherein the inference is predicted by the model operating on the node based on the sensor data collected by the portion of the one or more sensors,
wherein the inference represents an indication of an event, and
wherein the class, for each of the tuples, is one of multiple classes and is a trajectory class, which is determined from the sensor data;
storing the tuples in a first table;
generating a second table that associates the human operator with classes from the first table;
generating a prediction-action table that associates the inference to the operator action using a specific class from the second table, wherein the prediction-action table captures relationships between inferences and operator actions;
generating an efficiency score from the prediction-action table by comparing the inference and the operator action in each tuple;
determining an efficiency score distribution for the class;
assessing the quality of the performance of the model based on the efficiency score and the efficiency score distribution; and
retraining the model when the efficiency score distribution indicates that the human operator is not reacting in accordance with the inferences.
12 . The non-transitory storage medium of claim 11 , wherein the inference is Boolean and includes two types, and
wherein the operations further comprise generating a scenario efficiency for each of the two types.
13 . The non-transitory storage medium of claim 12 , wherein the operations further comprise weighting each of the scenario efficiencies and summing the weighted scenario efficiencies to determine the efficiency score.
14 . The non-transitory storage medium of claim 13 , wherein an efficiency score is generated for each of the classes represented in the second table.
15 . The non-transitory storage medium of claim 14 , wherein the operations further comprise generating the inference based on sensor data collected from sensors operating at the node.
16 . The non-transitory storage medium of claim 11 , wherein the operations further comprise training the model.
17 . The non-transitory storage medium of claim 11 , wherein the operations further comprise determining an efficiency score distribution for the model for each of the classes and assessing the quality of the performance of the model for each of the classes.
18 . The non-transitory storage medium of claim 17 , wherein the operations further comprise:
retraining the model for scenarios under a threshold efficiency score;
determining the class from sensor data generated at the node using a first model; and
determining the inference from a second model, wherein the class is the trajectory class,
wherein the model detects cornering events,
wherein the inference determines whether the cornering event is safe or dangerous, and
wherein the operator action is correlated to the trajectory class, the inference, and the operator identifier.