Language-based learning for monocular depth estimation
View Patent ↗Systems, methods, and other embodiments described herein relate to using a language model to facilitate training a depth model for monocular depth estimation. In one embodiment, a method includes acquiring an image depicting surrounding objects present in an environment. The method includes generating a depth map from the image using a depth model that performs monocular depth estimation. The method includes analyzing the depth map to derive a semantic loss according to a language model. The method includes training the depth model according to at least the semantic loss.
1 . A depth system, comprising:
one or more processors;
a memory communicably coupled to the one or more processors and storing instructions that, when executed by the one or more processors, cause the one or more processors to:
acquire an image depicting surrounding objects present in an environment;
generate a depth map from the image using a depth model that performs monocular depth estimation;
analyze the depth map to derive a semantic loss according to a language model, including querying the language model with a query including the depth map and a text string to identify anomalies in the depth map, wherein the text string defines at least one characteristic of the query; and
train the depth model according to at least the semantic loss.
2 . The depth system of claim 1 , wherein the depth model performs monocular depth estimation and is trained according to the semantic loss and a depth loss from self-supervised structure-from-motion (SfM) training.
3 . The depth system of claim 1 , wherein the language model is one of a large language model (LLM) and a visual language model (VLM), and
wherein the instructions to train the depth model include instructions to combine the semantic loss with a depth loss that is a self-supervised loss.
4 . The depth system of claim 1 , wherein the instructions to analyze the depth map include instructions to generate the semantic loss according to an output of the language model identifying a presence of one or more anomalies in the depth map.
5 . The depth system of claim 1 , wherein the instructions to analyze the depth map include instructions to generate a mask to segment anomalies within the depth map, and
wherein the instructions to train the depth model include instructions to filter the image from a training data set if the image causes anomalies in the depth map.
6 . The depth system of claim 1 , wherein the instructions include instructions to:
provide the depth model, including integrating the depth model in a perception pipeline of an autonomous vehicle to facilitate control of the autonomous vehicle.
7 . The depth system of claim 1 , wherein the depth system is embedded within a vehicle to perceive depth in the environment.
8 . A non-transitory computer-readable medium including instructions that, when executed by one or more processors, cause the one or more processors to:
acquire an image depicting surrounding objects present in an environment;
generate a depth map from the image using a depth model that performs monocular depth estimation;
analyze the depth map to derive a semantic loss according to a language model, including querying the language model with a query including the depth map and a text string to identify anomalies in the depth map, wherein the text string defines at least one characteristic of the query; and
train the depth model according to at least the semantic loss.
9 . The non-transitory computer-readable medium of claim 8 , wherein the depth model performs monocular depth estimation and is trained according to the semantic loss and a depth loss from self-supervised structure-from-motion (SfM) training.
10 . The non-transitory computer-readable medium of claim 8 , wherein the language model is one of a large language model (LLM) and a visual language model (VLM), and
wherein the instructions to train the depth model include instructions to combine the semantic loss with a depth loss that is a self-supervised loss.
11 . The non-transitory computer-readable medium of claim 8 , wherein the instructions to analyze the depth map include instructions to generate the semantic loss according to an output of the language model identifying a presence of one or more anomalies in the depth map.
12 . A method, comprising:
acquiring an image depicting surrounding objects present in an environment;
generating a depth map from the image using a depth model that performs monocular depth estimation;
analyzing the depth map to derive a semantic loss according to a language model, including querying the language model with a query including the depth map and a text string to identify anomalies in the depth map, wherein the text string defines at least one characteristic of the query; and
training the depth model according to at least the semantic loss.
13 . The method of claim 12 , wherein the depth model performs monocular depth estimation and is trained according to the semantic loss and a depth loss from self-supervised structure-from-motion (SfM) training.
14 . The method of claim 12 , wherein the language model is one of a large language model (LLM) and a visual language model (VLM), and
wherein training the depth model includes combining the semantic loss with a depth loss that is a self-supervised loss.
15 . The method of claim 12 , wherein analyzing the depth map includes generating the semantic loss according to an output of the language model identifying a presence of one or more anomalies in the depth map.
16 . The method of claim 12 , wherein analyzing the depth map includes generating a mask to segment anomalies within the depth map, and wherein training the depth model includes filtering the image from a training data set if the image causes anomalies in the depth map.
17 . The method of claim 12 , further comprising:
providing the depth model, including integrating the depth model in a perception pipeline of an autonomous vehicle to facilitate control of the autonomous vehicle.