Language-based learning for monocular depth estimation

1 . A depth system, comprising:

one or more processors;

a memory communicably coupled to the one or more processors and storing instructions that, when executed by the one or more processors, cause the one or more processors to:

acquire an image depicting surrounding objects present in an environment;

generate a depth map from the image using a depth model that performs monocular depth estimation;

analyze the depth map to derive a semantic loss according to a language model, including querying the language model with a query including the depth map and a text string to identify anomalies in the depth map, wherein the text string defines at least one characteristic of the query; and

train the depth model according to at least the semantic loss.

2 . The depth system of claim 1 , wherein the depth model performs monocular depth estimation and is trained according to the semantic loss and a depth loss from self-supervised structure-from-motion (SfM) training.

3 . The depth system of claim 1 , wherein the language model is one of a large language model (LLM) and a visual language model (VLM), and

wherein the instructions to train the depth model include instructions to combine the semantic loss with a depth loss that is a self-supervised loss.

4 . The depth system of claim 1 , wherein the instructions to analyze the depth map include instructions to generate the semantic loss according to an output of the language model identifying a presence of one or more anomalies in the depth map.

5 . The depth system of claim 1 , wherein the instructions to analyze the depth map include instructions to generate a mask to segment anomalies within the depth map, and

wherein the instructions to train the depth model include instructions to filter the image from a training data set if the image causes anomalies in the depth map.

6 . The depth system of claim 1 , wherein the instructions include instructions to:

provide the depth model, including integrating the depth model in a perception pipeline of an autonomous vehicle to facilitate control of the autonomous vehicle.

7 . The depth system of claim 1 , wherein the depth system is embedded within a vehicle to perceive depth in the environment.

8 . A non-transitory computer-readable medium including instructions that, when executed by one or more processors, cause the one or more processors to:

acquire an image depicting surrounding objects present in an environment;

generate a depth map from the image using a depth model that performs monocular depth estimation;

analyze the depth map to derive a semantic loss according to a language model, including querying the language model with a query including the depth map and a text string to identify anomalies in the depth map, wherein the text string defines at least one characteristic of the query; and

train the depth model according to at least the semantic loss.

9 . The non-transitory computer-readable medium of claim 8 , wherein the depth model performs monocular depth estimation and is trained according to the semantic loss and a depth loss from self-supervised structure-from-motion (SfM) training.

10 . The non-transitory computer-readable medium of claim 8 , wherein the language model is one of a large language model (LLM) and a visual language model (VLM), and

wherein the instructions to train the depth model include instructions to combine the semantic loss with a depth loss that is a self-supervised loss.

11 . The non-transitory computer-readable medium of claim 8 , wherein the instructions to analyze the depth map include instructions to generate the semantic loss according to an output of the language model identifying a presence of one or more anomalies in the depth map.

12 . A method, comprising:

acquiring an image depicting surrounding objects present in an environment;

generating a depth map from the image using a depth model that performs monocular depth estimation;

analyzing the depth map to derive a semantic loss according to a language model, including querying the language model with a query including the depth map and a text string to identify anomalies in the depth map, wherein the text string defines at least one characteristic of the query; and

training the depth model according to at least the semantic loss.

13 . The method of claim 12 , wherein the depth model performs monocular depth estimation and is trained according to the semantic loss and a depth loss from self-supervised structure-from-motion (SfM) training.

14 . The method of claim 12 , wherein the language model is one of a large language model (LLM) and a visual language model (VLM), and

wherein training the depth model includes combining the semantic loss with a depth loss that is a self-supervised loss.

15 . The method of claim 12 , wherein analyzing the depth map includes generating the semantic loss according to an output of the language model identifying a presence of one or more anomalies in the depth map.

16 . The method of claim 12 , wherein analyzing the depth map includes generating a mask to segment anomalies within the depth map, and wherein training the depth model includes filtering the image from a training data set if the image causes anomalies in the depth map.

17 . The method of claim 12 , further comprising:

providing the depth model, including integrating the depth model in a perception pipeline of an autonomous vehicle to facilitate control of the autonomous vehicle.