Modeling images obtained from vehicles using prompts

1 . A method comprising:

receiving, by one or more processors, an image from a camera of a vehicle, the camera recording activity performed by a person inside of the vehicle, the image depicting one or more objects;

receiving a prompt defining a target to be detected in the image;

generating a textual embedding based on the prompt;

analyzing the image and the textual embedding using a machine learning model that comprises a backbone model, to generate a prediction that indicates whether the one or more objects depicted in the image match the target defined by the prompt, wherein the textual embedding is aggregated with one or more layers of the backbone model to generate the prediction;

overlaying a bounding box on the image received from the camera of the vehicle, the bounding box enclosing a region of the image that corresponds with the one or more objects; and

presenting, at a client device associated with the vehicle, a message comprising the image that includes the bounding box that encloses the region of the image along with a set of instructions generated based on a type of the target defined by the prompt.

2 . The method of claim 1 , wherein the machine learning model comprises at least one of a Contrastive Language-Image Pre-Training (CLIP) neural network model or a CLIP-segmentation (CLIPSeg) neural network model, wherein the CLIP-segmentation model performs operations comprising:

translating images into image embeddings; and

generating segmentation masks using the image embeddings and the textual embeddings in accordance with M=f(Ei, Et), where M represents an estimated mask, Ei represents an image embedding, and Et represents the textual embedding.

3 . The method of claim 1 , wherein the prompt comprises text describing the target, the target described by the text corresponding to an operator of the vehicle holding an object in one hand while driving the vehicle, the message comprising a notification instructing the operator of the vehicle to stop holding the object in one hand while driving the vehicle.

4 . The method of claim 3 , further comprising:

solving an optimization problem corresponding to a loss function defined as argminE L(f(Ei, Et), M) to generate an embedding that represents a minimal amount of loss between a predicted heat map and a ground truth mask, where the optimization adjusts the text embedding until a stopping criterion is satisfied.

5 . The method of claim 1 , wherein the prompt comprises a mask defining the target, the message indicating that the target defined by the prompt has been detected and information on how to resolve the target.

6 . The method of claim 5 , further comprising:

generating an embedding representing the mask, wherein the data associated with the prompt comprises the embedding.

7 . The method of claim 6 , wherein generating the embedding comprises:

receiving an annotated training image depicting a training object corresponding to the target;

receiving a text-based prompt defining the target; and

analyzing the annotated training image and the text-based prompt using the machine learning model to perform operations comprising:

translating the annotated training image into an image embedding;

translating the text-based prompt into a text embedding; and

estimating an output mask identifying a region of the annotated training image corresponding to the target using the image embedding and the text embedding.

8 . The method of claim 7 , further comprising:

computing a loss function based on the estimated output mask and a ground truth mask; and

updating the text embedding to regenerate the output mask and recompute the loss function until a stopping criterion is satisfied, the mask being generated corresponding to the estimated output mask.

9 . The method of claim 8 , further comprising:

receiving a second annotated training image depicting a training object that excludes the target; and

analyzing the second annotated training image and the text-based prompt using the machine learning model to perform operations comprising:

translating the second annotated training image into a second image embedding; and

estimating a second output mask identifying a region of the second annotated training image corresponding to the target using the second image embedding and the text embedding.

10 . The method of claim 9 , further comprising:

re-computing the loss function based on the second output mask and the ground truth mask; and

updating the text embedding to regenerate the second output mask and recompute the loss function until the stopping criterion is satisfied, the loss function being computed based on solving an optimization problem.

11 . The method of claim 1 , wherein the prompt defining the target is different for different types of vehicles.

12 . The method of claim 11 , further comprising:

determining a type of vehicle associated with the camera; and

selecting a prompt from a plurality of prompts based on the type of vehicle that is determined, the image and data associated with the prompt being analyzed based on the selected prompt.

13 . The method of claim 1 , wherein the prediction generated by the machine learning model comprises a mask that identifies a region of the image corresponding to the target, further comprising:

processing the image to generate a heat map that includes pixels of different gradients indicating differing likelihoods of the target being present in the image; and

processing the heat map to generate the bounding box around the target, the heat map being presented to an operator in response to determining that a certain quantity of pixels transgresses a likelihood threshold.

14 . The method of claim 1 , wherein the message comprises a heat map representing pixels of the image that correspond to the target, the heat map being presented to an operator of the vehicle, the heat map including pixels of different gradients indicating differing likelihoods of the target being present in the image, the heat map indicating pixel values that correspond to a likelihood above a minimum threshold of the target being present.

15 . The method of claim 14 , wherein the message comprises the bounding box representing pixels of the image that correspond to the target, the image being processed together with the heat map to generate the bounding box around the region of the image that is associated with the pixel values above the minimum threshold.

16 . The method of claim 1 , wherein the data associated with the prompt comprises a one-hot vector or floating point vector used as a task embedding for the machine learning model.

17 . The method of claim 1 , further comprising:

generating a heat map based on the prediction; and

generating a bounding box based on the heat map.

18 . The method of claim 17 , further comprising:

comparing a size of the bounding box to a threshold; and

conditioning generation of the message based on whether the size of the bounding box transgresses the threshold.

19 . A vehicle system comprising:

one or more computer processors; and

one or more non-transitory computer-readable mediums storing instructions that, when executed by the one or more computer processors, cause the vehicle system to perform operations comprising:

receiving an image from a camera of a vehicle, the camera recording activity performed by a person inside of the vehicle, the image depicting one or more objects;

receiving a prompt defining a target to be detected in the image;

generating a textual embedding based on the prompt;

analyzing the image and the textual embedding using a machine learning model that comprises a backbone model, to generate a prediction that indicates whether the one or more objects depicted in the image match the target defined by the prompt, wherein the textual embedding is aggregated with one or more layers of the backbone model to generate the prediction;

overlaying a bounding box on the image received from the camera of the vehicle, the bounding box enclosing a region of the image that corresponds with the one or more objects; and

presenting, at a client device associated with the vehicle, a message comprising the image that includes the bounding box that encloses the region of the image along with a set of instructions generated based on a type of the target defined by the prompt.

20 . A non-transitory computer-readable medium storing instructions that, when executed by one or more computer processors of a vehicle system, cause the vehicle system to perform operations comprising:

receiving an image from a camera of a vehicle, the camera recording activity performed by a person inside of the vehicle, the image depicting one or more objects;

receiving a prompt defining a target to be detected in the image;

generating a textual embedding based on the prompt;

analyzing the image and the textual embedding using a machine learning model that comprises a backbone model, to generate a prediction that indicates whether the one or more objects depicted in the image match the target defined by the prompt, wherein the textual embedding is aggregated with one or more layers of the backbone model to generate the prediction;

overlaying a bounding box on the image received from the camera of the vehicle, the bounding box enclosing a region of the image that corresponds with the one or more objects; and

presenting, at a client device associated with the vehicle, a message comprising the image that includes the bounding box that encloses the region of the image along with a set of instructions generated based on a type of the target defined by the prompt.