Virtual models for communications between user devices and external observers
Systems and methods for interactions between an autonomous vehicle and one or more external observers include virtual models of drivers the autonomous vehicle. The virtual models may be generated by the autonomous vehicle and displayed to one or more external observers, and in some cases using devices worn by the external observers. The virtual models may facilitate interactions between the external observers and the autonomous vehicle using gestures or other visual cues. The virtual models may be encrypted with characteristics of an external observer, such as the external observer's face image, iris, or other representative features. Multiple virtual models for multiple external observers may be simultaneously used for multiple communications while preventing interference due to possible overlap of the multiple virtual models.
1. A user device, comprising:
an image sensor;
a display visible to a first external observer;
a memory; and
a processor configured to:
obtain a first one or more images from the image sensor, the first one or more images including at least a portion of the first external observer relative to an exterior of the user device;
determine, based on one or more features in the first one or more images, that the first external observer is in a presence of an external environment of the user device;
output, to the display based on the determination that the first external observer is in the presence of the external environment of the user device, a first virtual model observable by the first external observer, the first virtual model depicting a user of the user device;
after the first virtual model is output to the display based on the determination that the first external observer is in the presence of the external environment of the user device, obtain a second one or more images from the image sensor, the second one or more images not including at least a portion of the first external observer;
determine, based on the second one or more images, that the first external observer is not in the presence of the external environment of the user device; and
disable the output of the first virtual model based on the determination that the first external observer is not in the presence of the external environment of the user device.
2. The user device of claim 1 , wherein the processor is further configured to:
identify an input associated with the first external observer; and
determine that the first external observer is in the presence of the external environment of the user device based on the input.
3. The user device of claim 2 , wherein the input includes one or more gestures from the first external observer.
4. The user device of claim 2 , wherein the input includes a voice input from the first external observer.
5. The user device of claim 1 , wherein, to determine that the first external observer is in the presence of the external environment of the user device, the processor is configured to:
track, based on the one or more features in the first one or more images, an eye gaze of the first external observer.
6. The user device of claim 5 , wherein the processor is configured to:
determine a field of view of the first external observer based on tracking the eye gaze; and
determine that the field of view includes at least a portion of the user device.
7. The user device of claim 1 , wherein the image sensor includes at least one of a camera or a depth sensor.
8. The user device of claim 1 , wherein the processor is further configured to:
determine that a second external observer is in the presence of the external environment of the user device; and
output, to the display based on the determination that the second external observer is in the presence of the external environment of the user device, the first virtual model observable by the second external observer.
9. The user device of claim 1 , wherein the processor is further configured to:
determine that a second external observer is in the presence of the external environment of the user device; and
output, to the display based on the determination that the second external observer is in the presence of the external environment of the user device, a second virtual model observable by the second external observer.
10. The user device of claim 1 , wherein the first virtual model is a visual depiction of a portion of the user of the user device.
11. The user device of claim 1 , wherein, to output the first virtual model, the processor is further configured to output audio associated with the first virtual model.
12. The user device of claim 1 , wherein the processor is configured to:
communicate with the first external observer using the first virtual model in response to a communication from the first external observer.
13. The user device of claim 12 , wherein the communication from the first external observer includes one or more gestures.
14. The user device of claim 1 , wherein the user device includes a head mounted display, a virtual reality device, an augmented reality device, or a vehicle.
15. A method of communicating with a first external observer, comprising:
obtaining, by a user device, a first one or more images including at least a portion of the first external observer relative to an exterior of the user device;
determining, based on one or more features in the first one or more images, that the first external observer is in a presence of an external environment of the user device;
displaying, based on the determination that the first external observer is in the presence of the external environment of the user device, a first virtual model observable by the first external observer, the first virtual model depicting a user of the user device;
after the first virtual model is displayed based on the determination that the first external observer is in the presence of the external environment of the user device, obtaining a second one or more images, the second one or more images not including at least a portion of the first external observer;
determining, based on the second one or more images, that the first external observer is not in the presence of the external environment of the user device; and
disabling the display of the first virtual model based on the determination that the first external observer is not in the presence of the external environment of the user device.
16. The method of claim 15 , further comprising:
identifying an input associated with the first external observer; and
determining that the first external observer is in the presence of the external environment of the user device based on the input.
17. The method of claim 16 , wherein the input includes one or more gestures from the first external observer.
18. The method of claim 16 , wherein the input includes a voice input from the first external observer.
19. The method of claim 15 , wherein determining that the first external observer is in the presence of the external environment of the user device comprises:
tracking, based on the one or more features in the first one or more images, an eye gaze of the first external observer.
20. The method of claim 19 , further comprising:
determining a field of view of the first external observer based on tracking the eye gaze; and
determining that the field of view includes at least a portion of the user device.
21. The method of claim 15 , wherein the first one or more images are obtained from an image sensor, and wherein the image sensor includes at least one of a camera or a depth sensor.
22. The method of claim 15 , further comprising:
determining that a second external observer is in the presence of the external environment of the user device; and
displaying, based on the determination that the second external observer is in the presence of the external environment of the user device, the first virtual model observable by the second external observer.
23. The method of claim 15 , further comprising:
determining that a second external observer is in the presence of the external environment of the user device; and
displaying, based on the determination that the second external observer is in the presence of the external environment of the user device, a second virtual model observable by the second external observer.
24. The method of claim 15 , wherein the first virtual model is a visual depiction of a portion of the user of the user device.
25. The method of claim 15 , wherein outputting the first virtual model comprises outputting audio associated with the first virtual model.
26. The method of claim 15 , further comprising:
communicating with the first external observer using the first virtual model in response to a communication from the first external observer.
27. The method of claim 26 , wherein the communication from the first external observer includes one or more gestures.
28. The method of claim 15 , wherein the user device includes a head mounted display, a virtual reality device, an augmented reality device, or a vehicle.
29. The user device of claim 1 , wherein:
to determine that the first external observer is not in the presence of the external environment of the user device, the processor is configured to determine that no external observer is in the presence of the external environment of the user device; and
disable the output of the first virtual model based on the determination that no external observer is in the presence of the external environment of the user device.
30. The method of claim 15 , wherein:
determining that the first external observer is not in the presence of the external environment of the user device comprises determining that no external observer is in the presence of the external environment of the user device; and
the display of the first virtual model is disabled based on the determination that no external observer is in the presence of the external environment of the user device.