Apparatus, system and method for translating sensor label data between sensor domains
Technologies and techniques for converting sensor data, used in a vehicle or other device. A machine-learning model is applied to first sensor data, including a first operational characteristic capability and first sensor label data, wherein the machine-learning model is trained to second sensor data including a second operational characteristic capability. New sensor data is generated that corresponds to the applied machine-learning model, wherein the new sensor data includes translated first sensor label data. A loss function may be applied to the new sensor data to determine the accuracy of the new sensor data and translated first sensor label data. In some examples, a multi-dimensional matrix of camera sensor parameters may be applied to the first sensor data labels to transform the first sensor data labels to second sensor data labels.
1. A method of converting sensor data for autonomous vehicle perception, comprising:
receiving, by a processing system of an autonomous vehicle, first sensor data from a first sensor, wherein the first sensor data comprises a first operational characteristic capability and first sensor label data comprising at least one of semantic segmentation labels or bounding box masks;
applying, by the processing system, a machine-learning model comprising an encoder-decoder network to the first sensor data, wherein the machine-learning model is trained to translate the first sensor data to emulate second sensor data comprising a second operational characteristic capability different from the first operational characteristic capability;
generating, by the processing system, new sensor data corresponding to the applied machine-learning model, wherein the new sensor data comprises translated first sensor label data emulating the second operational characteristic capability; and
applying, by the processing system, a loss function to the new sensor data to validate the accuracy of the new sensor data and translated first sensor label data; and
configuring the processing system of the autonomous vehicle to perform object detection using the validated new sensor data and translated first sensor label data.
2. The method of claim 1 , wherein applying the machine-learning model comprises applying a deep neural network (DNN).
3. The method of claim 2 , wherein applying the DNN comprises processing the first sensor data via an encoder-decoder network and a generative adversarial network.
4. The method of claim 1 , wherein applying the loss function to the new sensor data comprises applying a consistency loss function to the translated first sensor label data relative to the first sensor label data.
5. The method of claim 1 , wherein applying the loss function to the new sensor data comprises
applying a second machine-learning model to the new sensor data, wherein the second machine-learning model is trained to the first sensor label data to produce modified new sensor label data, and wherein the second machine-learning model comprises a deep convolutional neural network (DNN).
6. The method of claim 5 , wherein the DNN comprises an encoder-decoder network and a generative adversarial network, configured to process the modified new sensor data in the opposite direction of the first machine learning model.
7. The method of claim 1 , wherein the machine-learning model is trained to translate the first sensor data to emulate second sensor data in an unpaired data environment.
8. A method of converting sensor label data for autonomous vehicle perception, comprising:
receiving, by a processing system of an autonomous vehicle, first sensor data from a first sensor, wherein the first sensor data comprises the first operational characteristic capability and first sensor label data comprising at least one of semantic segmentation labels or bounding box masks;
receiving, by the processing system, second sensor data from a second sensor, wherein the second sensor data comprises a second operational characteristic capability different from the first operational characteristic capability;
applying, by the processing system, a machine-learning model comprising an encoder-decoder network to the first sensor data and second sensor data, wherein the machine-learning model is trained to infer first sensor label data to the second sensor data;
generating, by the processing system, new sensor data corresponding to the applied machine-learning model, wherein the new sensor data comprises the inferred first sensor label data emulating the second operational characteristic capability; and
configuring the processing system of the autonomous vehicle to perform object detection using the new sensor data and inferred first sensor label data.
9. The method of claim 8 , further comprising applying a loss function to the new sensor data to determine the accuracy of the inferred first sensor label data relative to the first sensor label data.
10. The method of claim 9 , wherein the loss function comprises a consistency loss function.
11. The method of claim 8 , wherein applying the machine-learning model comprises applying a deep neural network to the first sensor data, second sensor data, and first sensor label data.
12. The method of claim 8 , wherein applying the machine-learning model to infer first sensor label data comprises applying extrinsic parameters of camera matrix data.
13. The method of claim 12 , wherein applying the extrinsic parameters comprises transferring sensor labels to the new sensor data.
14. The method of claim 8 , wherein the second sensor data is translated from the first sensor data via a generative adversarial network.
15. A method for translating sensor data for autonomous vehicle perception, comprising:
receiving, by a processing system of an autonomous vehicle, first sensor data from a first sensor, wherein the first sensor data comprises a first operational characteristic capability and first sensor label data comprising at least one of semantic segmentation labels or bounding box masks;
receiving, by the processing system, second sensor data from a second sensor, wherein the second sensor data comprises a second operational characteristic capability different from the first operational characteristic capability; and
applying, by the processing system, a machine-learning model comprising an encoder-decoder network to the first sensor data and second sensor data in an unpaired data environment, wherein the machine-learning model is trained to translate the first sensor label data to emulate the second operational characteristic capability;
generating, by the processing system, new sensor data corresponding to the applied machine-learning model, wherein the new sensor data comprises translated first sensor label data emulating the second operational characteristic capability;
applying, by the processing system, a loss function to the new sensor data to validate the accuracy of the new sensor data and translated first sensor label data; and
configuring the processing system of the autonomous vehicle to perform object detection using the validated new sensor data and translated first sensor label data.
16. The method of claim 15 , wherein the camera sensor parameters comprise intrinsic and extrinsic coefficients representing three-dimensional world points and their corresponding two-dimensional image points.
17. The method of claim 16 , wherein the extrinsic parameters represent a location of a first and second sensor in a three-dimensional scene, and the intrinsic parameters represent an optical center and focal length of the first and second sensor.
18. The method of claim 15 , wherein translating the first sensor data to the second sensor data comprises applying a color mapping transformation to adjust coloration differences between the first sensor data and the second sensor data.
19. The method of claim 15 , wherein translating the first sensor data to the second sensor data comprises adjusting for differences in temporal sampling intervals between the first sensor and the second sensor.
20. The method of claim 15 , wherein the machine-learning algorithm comprises a reinforcement learning model configured to refine the accuracy of the transformed sensor data and associated label data using feedback derived from a validation process.