Methods and apparatuses for operating a self-driving vehicle

1. A method for operating a self-driving vehicle, comprising:

receiving one or more images having a plurality of objects;

generating a plurality of visually-descriptive latent vectors based on the one or more images;

receiving a notification from an occupant of the self-driving vehicle;

generating a latent vector based on the notification;

generating a feature vector based on the latent vector and at least one of the plurality of visually-descriptive latent vectors;

generating an attention map highlighting at least some portions of the plurality of objects based on the feature vector; and

providing at least one of a steering control or a velocity control to operate the self-driving vehicle based on the attention map and the notification.

2. The method of claim 1 , further comprises preprocessing the one or more images by performing at least one of a down-sampling, resizing, or normalization.

3. The method of claim 1 , wherein the steering control includes controlling a steering angle of a steering wheel of the self-driving vehicle and the velocity control includes a controlling a force applied to an accelerator of the self-driving vehicle.

4. The method of claim 1 , further comprises applying the one or more images to a cellular neural network to identify the plurality of objects.

5. The method of claim 1 , wherein the notification is a goal-oriented verbal notification or a stimulus-driven verbal notification.

6. The method of claim 5 , further comprises, prior to generating the attention map, performing a speech-to-text conversion to generate a textual representation of the notification.

7. A self-driving vehicle, comprising:

a memory;

one or more processors communicatively coupled to the memory, the one or more processors perform the steps of:

receiving one or more images having a plurality of objects;

generating a plurality of visually-descriptive latent vectors based on the one or more images;

receiving a notification from an occupant of the self-driving vehicle;

generating a latent vector based on the notification;

generating a feature vector based on the latent vector and at least one of the plurality of visually-descriptive latent vectors;

generating an attention map highlighting at least some portions of the plurality of objects based on the feature vector; and

providing at least one of a steering control or a velocity control to control an operation of the vehicle based on the attention map and the notification.

8. The vehicle of claim 7 , wherein the one or more processors further perform the steps of preprocessing the one or more images by performing at least one of a down-sampling, resizing, or normalization.

9. The vehicle of claim 7 , wherein the steering control includes controlling a steering angle of a steering wheel of the self-driving vehicle and the velocity control includes a controlling a force applied to an accelerator of the self-driving vehicle.

10. The vehicle of claim 7 , wherein the one or more processors further perform the steps of applying the one or more images to a cellular neural network to identify the plurality of objects.

11. The vehicle of claim 7 , wherein the notification is a goal-oriented verbal notification or a stimulus-driven verbal notification.

12. The vehicle of claim 11 , wherein the one or more processors further perform the steps of, prior to generating the attention map, performing a speech-to-text conversion to generate a textual representation of the notification.

13. A non-transitory computer readable medium having instructions stored therein, the instructions, when executed by one or more processors of the self-driving vehicle, cause the one or more processors to:

receive one or more images having a plurality of objects;

generate a plurality of visually-descriptive latent vectors based on the one or more images;

receive a notification from an occupant of the self-driving vehicle;

generate a latent vector based on the notification;

generate a feature vector based on the latent vector and at least one of the plurality of visually-descriptive latent vectors;

generate an attention map highlighting at least some portions of the plurality of objects based on the feature vector; and

provide at least one of a steering control or a velocity control to control an operation of the self-driving vehicle based on the attention map and the notification.

14. The non-transitory computer readable medium of claim 13 , further comprises instructions, when executed by the one or more processors of the self-driving vehicle, cause the one or more processors to perform at least one of a down-sampling, resizing, or normalization.

15. The non-transitory computer readable medium of claim 13 , wherein the steering control includes controlling a steering angle of a steering wheel of the self-driving vehicle and the velocity control includes a controlling a force applied to an accelerator of the self-driving vehicle.

16. The non-transitory computer readable medium of claim 13 , further comprises instructions, when executed by the one or more processors of the self-driving vehicle, cause the one or more processors to apply the one or more images to a cellular neural network to identify the plurality of objects.

17. The non-transitory computer readable medium of claim 13 , wherein the notification is a goal-oriented verbal notification or a stimulus-driven verbal notification.

18. The computer readable medium of claim 17 , further comprises instructions, when executed by the one or more processors of the self-driving vehicle, cause the one or more processors to, prior to generating the attention heatmap, perform a speech-to-text conversion to generate a textual representation of the notification.