Mixed-batch training of a multi-task network

1. A perception system, comprising:

one or more processors; and

a memory communicably coupled to the one or more processors and storing:

a network module including instructions that, when executed by the one or more processors, cause the one or more processors to:

acquire training data that includes real data and virtual data for training a multi-task network that performs at least depth prediction and semantic segmentation, the virtual data including synthetic images;

generate a first output from the multi-task network using the real data and second output from the multi-task network using the virtual data over separate executions of the multi-task network;

generate a mixed loss by analyzing the first output to produce a real loss that includes at least a self-supervised loss and the second output to produce a virtual loss that includes a supervised loss comprised of a semantic loss, a depth loss, a surface normal loss, and a synthesis loss; and

update the multi-task network using the mixed loss.

2. The perception system of claim 1 , wherein the network module includes instructions to analyze the first output to produce the real loss including instructions to generate the real loss as the self-supervised loss using a pair of monocular images that are part of the real data.

3. The perception system of claim 1 , wherein the network module includes instructions to analyze the second output to produce the virtual loss including instructions to:

produce the semantic loss by applying a cross-entropy loss between ground truth semantic labels and a semantic map of the second output,

produce the depth loss by applying a scale-invariant logarithmic loss (SILog) between a depth map of the second output and a ground-truth depth map from the training data,

produce the surface normal loss by a similarity metric to surface normals derived from the second output and a ground-truth normal from the training data, and

produce the synthesis loss by applying a partially supervised photometric loss to the second output.

4. The perception system of claim 1 , wherein the network module includes instructions to generate the mixed loss including instructions to combine the real loss and the virtual loss according to a weighted coefficient, and

wherein the virtual data includes ground-truth data for depth and semantics corresponding to a monocular image.

5. The perception system of claim 1 , wherein the network module includes instructions to generate the first output and the second output using the multi-task network including instructions to process the training data using an image encoder of the multi-task network to generate image features and decode the image features into a semantic map using a semantic decoder and a depth map using a depth decoder of the multi-task network, and

wherein a pose encoder of the multi-task network generates a rigid-body transformation between real images in the training data.

6. The perception system of claim 1 , wherein the network module includes instructions to provide a depth map and a semantic map using the multi-task network to control a device to navigate through a surrounding environment.

7. The perception system of claim 1 , wherein the perception system is integrated with a vehicle that operates autonomously.

8. A non-transitory computer-readable medium including instructions that, when executed by one or more processors, cause the one or more processors to:

acquire training data that includes real data and virtual data for training a multi-task network that performs at least depth prediction and semantic segmentation, the virtual data including synthetic images;

generate a first output from the multi-task network using the real data and second output from the multi-task network using the virtual data over separate executions of the multi-task network;

generate a mixed loss by analyzing the first output to produce a real loss that includes at least a self-supervised loss and the second output to produce a virtual loss that includes a supervised loss comprised of a semantic loss, a depth loss, a surface normal loss, and a synthesis loss; and

update the multi-task network using the mixed loss.

9. The non-transitory computer-readable medium of claim 8 , wherein the instructions to analyze the first output to produce the real loss include instructions to generate the real loss as the self-supervised loss using a pair of monocular images that are part of the real data.

10. The non-transitory computer-readable medium of claim 8 , wherein the instructions to generate the mixed loss include instructions to combine the real loss and the virtual loss according to a weighted coefficient, and

wherein the virtual data includes ground-truth data for depth and semantics corresponding to a monocular image.

11. The non-transitory computer-readable medium of claim 8 , wherein the instructions to generate the first output and the second output using the multi-task network include instructions to process the training data using an image encoder of the multi-task network to generate image features and decode the image features into a semantic map using a semantic decoder and a depth map using a depth decoder of the multi-task network, and

wherein a pose encoder of the multi-task network generates a rigid-body transformation between real images in the training data.

12. A method, comprising:

acquiring training data that includes real data and virtual data for training a multi-task network that performs at least depth prediction and semantic segmentation, the virtual data including synthetic images;

generating a first output from the multi-task network using the real data and a second output from the multi-task network using the virtual data over separate executions of the multi-task network;

generating a mixed loss by analyzing the first output to produce a real loss that includes at least a self-supervised loss and the second output to produce a virtual loss that includes a supervised loss comprised of a semantic loss, a depth loss, a surface normal loss, and a synthesis loss; and

updating the multi-task network using the mixed loss.

13. The method of claim 12 , wherein analyzing the first output to produce the real loss includes generating the real loss as the self-supervised loss using a pair of monocular images that are part of the real data.

14. The method of claim 12 , wherein analyzing the second output to produce the virtual loss includes:

producing the semantic loss by applying a cross-entropy loss between ground truth semantic labels and a semantic map of the second output,

producing the depth loss by applying a scale-invariant logarithmic loss (SILog) between a depth map of the second output and a ground-truth depth map from the training data,

producing the surface normal loss by a similarity metric to surface normals derived from the second output and a ground-truth normal from the training data, and

producing the synthesis loss by applying a partially supervised photometric loss to the second output.

15. The method of claim 12 , wherein generating the mixed loss includes combining the real loss and the virtual loss according to a weighted coefficient, and

wherein the virtual data includes ground-truth data for depth and semantics corresponding to a monocular image.

16. The method of claim 12 , wherein generating the first output and the second output using the multi-task network includes processing the training data using an image encoder of the multi-task network to generate image features and decoding the image features into a semantic map using a semantic decoder and a depth map using a depth decoder of the multi-task network, and

wherein a pose encoder of the multi-task network generates a rigid-body transformation between real images in the training data.

17. The method of claim 12 , further comprising:

providing a depth map and a semantic map using the multi-task network to control a device to navigate through a surrounding environment.