Real-time validation of robotic sensing systems
Disclosed herein are systems, devices, and apparatuses for improved perception/sensing systems in robots or other vehicles. The system receives sensor data representative of a field of view of a robot and determines, based on the sensor data and an object detection model, an identification of an object within the field of view and an accuracy metric of the identification of the object, wherein the object detection model relates the sensor data to the identification. The system also requests, based on the accuracy metric, an informational feedback from the identification of the object and updates the object detection model to an updated object detection model based on the informational feedback.
1 . A device comprising a processor configured to:
receive sensor data representative of a field of view of a robot;
determine, based on the sensor data and an object detection model, an identification of an object within the field of view and an accuracy metric of the identification of the object, wherein the object detection model relates the sensor data to the identification, wherein the sensor data comprises image data comprising a series of frames within the field of view;
generate detection information associated with the identification of the object, wherein the detection information comprises: a frame number of a frame within the series of frames in which the object is detected; a location of the object within the frame; and the accuracy metric;
request, based on the accuracy metric, an informational feedback regarding the identification of the object, wherein the informational feedback comprises an identification label for the object;
generate, from the identification label for the object and the detection information, training information for training the object detection model based on the identification label and the detection information, wherein the training information comprises the identification label, the frame number, the location, and the accuracy metric; and
train the object detection model based on the training information to obtain an updated object detection model.
2 . The device of claim 1 , wherein the processor is further configured to control movements of the robot based on the updated object detection model.
3 . The device of claim 1 , wherein the processor is further configured to request the informational feedback based on whether the accuracy metric satisfies a predefined criterion.
4 . The device of claim 3 , wherein the predefined criterion comprises whether the accuracy metric is below a threshold value of accuracy.
5 . The device of claim 1 , wherein the informational feedback comprises a verbal feedback or an electronic message received from a nearby human or from a second robot.
6 . The device of claim 1 , wherein the accuracy metric comprises a confidence metric indicating an extent to which the identification of the object substantially correct.
7 . The device of claim 1 , wherein the processor is further configured to:
maintain a representation of an operational environment of the robot, wherein the representation comprises detected objects in the operational environment, wherein the object comprises one of the detected objects, wherein the processor is further configured to request, based on the accuracy metric, the informational feedback about the one of the detected objects; and
update the representation of the operational environment to an updated representation based on the informational feedback about the one of the detected objects.
8 . The device of claim 1 , wherein the robot is configured to operate within an operational design domain, wherein the processor is further configured to:
determine, based on the sensor data, an operation status indicating whether the robot is operating within the operational design domain;
request, based on the operational status, an operational feedback with respect to the operational status; and
update the operational design domain to an updated operational design domain that is based on the operational feedback.
9 . The device of claim 1 , wherein the sensor data comprises image data of the field of view, wherein the processor is further configured to superimpose the accuracy metric and an identification label of the object into the image data to generate an augmented image data.
10 . The device of claim 9 , wherein the device is further configured to transmit the augmented image data to an external display device.
11 . The device of claim 1 , wherein the updated object detection model comprises a higher accuracy metric for the identification of the object as compared to the accuracy metric of the object detection model for the identification of the object.
12 . A device comprising:
a sensor configured to capture sensor data within a field of view of a robot, wherein the sensor data comprises image data comprising a series of frames within the field of view;
a processor configured to:
perform an identification of a detected object within the field of view of the robot, wherein the identification is output from an object detection model based on the sensor data; and
determine an accuracy metric associated with the identification that is output from the object detection model;
generate detection information associated with the identification of the detected object, wherein the detection information comprises: a frame number of a frame within the series of frames in which the detected object is detected; a location of the detected object within the frame; and the accuracy metric; and
an audio interface communicatively coupled with the processor, the audio interface configured to request a verbal feedback regarding the identification of the detected object, wherein the processor is configured to update the object detection model into an updated object detection model based on the verbal feedback, wherein the processor configured to update the object detection model into the updated object detection model based on the verbal feedback comprises the processor further configured to:
generate, from the verbal feedback and the detection information, training information for training the object detection model based on the verbal feedback and the detection information, wherein the training information comprises the verbal feedback regarding the identification, the frame number, the location, and the accuracy metric; and
train the object detection model based on the training information to obtain the updated object detection model.
13 . The device of claim 12 , the device further comprising a control system configured to control movements of the robot based on the updated object detection model.
14 . The device of claim 12 , wherein the audio interface comprises a microphone for receiving the verbal feedback and a speaker configured to audibly provide a request for the verbal feedback.