Spatial audio for wayfinding
The technology employs spatial audio information to enhance wayfinding for pickup, drop-off and in-vehicle situations. The spatial information has a directional component, and a sense of distance can also be incorporated into the audio information. Audio cues or other spatial information is provided via headphones worn by a user. The spatial audio gives the user direction information, which can help locate the vehicle. In addition, this approach can be used when the rider is in the vehicle prior to exiting. For instance, spatial audio can be provided to the rider to give them contextual information about the environment outside the vehicle prior to exiting, such as whether a bicyclist is approaching on the side they will be exiting. This contextual information can alert the rider to wait or otherwise be more situationally aware when departing the vehicle.
1 . A computer-implemented method, comprising:
estimating, by one or more processors, a current location of a user;
identifying, by the one or more processors, an actual location of a vehicle;
estimating, by the one or more processors, a pose of the user based on dynamic detection of an orientation of the user's head;
selecting, by the one or more processors, a sound;
generating, by the one or more processors based on the selected sound, a spatial audio signal, the spatial audio signal having a directional component according to a heading of the user and a heading of the vehicle, and the generating including dynamically adjusting the directional component of the spatial audio signal in response to detected changes in the estimated pose of the user; and
providing the spatial audio signal for presentation to the user.
2 . The method of claim 1 , wherein the detected changes in the estimated pose of the user includes dynamic detection of the orientation of the user's head based on detecting head movements for determining a looking direction of the user and detecting body movements for determining a movement direction of the user.
3 . The method of claim 1 , wherein the dynamic detection of the orientation of the user's head is based on sensor information obtained from at least one sensor of headphones worn by the user, and the at least one sensor includes at least one of a gyroscope of the headphones, a magnetometer of the headphones, or an accelerometer of the headphones.
4 . The method of claim 1 , wherein estimating the current location of the user is based on sensor information received from headphones worn by the user or based on sensor information received from a client device of the user.
5 . The method of claim 1 , wherein providing the spatial audio signal for presentation to the user includes transmitting the spatial audio signal from the vehicle or a client device of the user to headphones worn by the user, and estimating the pose of the user is based on sensor information obtained by the client device.
6 . The method of claim 1 , wherein the heading of the vehicle is based on sensor information from a positioning system of the vehicle.
7 . The method of claim 1 , wherein estimating the pose of the user is based on perception information received from the vehicle.
8 . The method of claim 1 , wherein the spatial audio signal further includes a loudness component to indicate a relative distance between the current location of the user and the actual location of the vehicle.
9 . The method of claim 1 , wherein generating the spatial audio signal includes generating a three-dimensional scene based on the current location of the user or the actual location of the vehicle, and the three-dimensional scene includes one or more objects in an environment of the user.
10 . The method of claim 1 , wherein generating the spatial audio signal further includes generating the spatial audio signal having the directional component according to the estimated pose of the user relative to a new location of the vehicle.
11 . The method of claim 1 , further comprising varying a directional accuracy of the spatial audio signal based on a relative distance between the user and the actual location of the vehicle.
12 . The method of claim 1 , further comprising generating turn-by-turn directions for the user to the vehicle, the spatial audio signal being configured to play concurrently with the turn-by-turn directions.
13 . The method of claim 1 , wherein the sound is selected based, at least partially on ambient noise around the user.
14 . A computing device, comprising:
memory storing a set of sounds; and
one or more processors operatively coupled to the memory, the one or more processors being configured to:
estimate a current location of a user;
identify an actual location of a vehicle;
estimate a pose of the user based on dynamic detection of an orientation of the user's head;
select a sound from the set of sounds in the memory;
generate, based on the selected sound, a spatial audio signal, the spatial audio signal having a directional component according to a heading of the user and a heading of the vehicle, and the generation of the spatial audio signal includes dynamically adjusting the directional component of the spatial audio signal in response to detected changes in the estimated pose of the user; and
provide the spatial audio signal for presentation to the user.
15 . A computer-implemented method, comprising:
identifying, by one or more processors of a vehicle, an actual location of the vehicle;
estimating, by the one or more processors, a heading of the vehicle;
detecting, by a perception system of the vehicle, a presence of an object in a vicinity of the vehicle;
selecting, by the one or more processors, a sound based on detecting the presence of the object in the vicinity of the vehicle;
generating, by the one or more processors, based on the selected sound, a spatial audio signal, the spatial audio signal having a directional component according to the heading of the vehicle, a relative position of the object in the vicinity of the vehicle, and dynamic detection of an orientation of a rider's head, the generating includes dynamically adjusting the directional component of the spatial audio signal in response to detected changes in the dynamic detection of the orientation of the rider's head; and
providing, by the one or more processors, the spatial audio signal for presentation to the rider within the vehicle.
16 . The method of claim 15 , wherein the sound is selected, at least partially, based on a user preference.
17 . The method of claim 15 , wherein the spatial audio signal indicates a side of the vehicle to exit or a door of the vehicle to exit.
18 . The method of claim 15 , wherein the heading of the vehicle is based on sensor information from a positioning system of the vehicle.
19 . The method of claim 15 , wherein the spatial audio signal further includes a loudness component to indicate a relative distance between the actual location of the vehicle and a location of the object in the vicinity of the vehicle, and the method further comprises varying the loudness component based on the relative distance between the object and the vehicle.
20 . The method of claim 15 , wherein generating the spatial audio signal further includes generating the spatial audio signal having the directional component according to the dynamic detection of the orientation of the rider's head relative to a new location of the vehicle or a new heading of the vehicle.
21 . The method of claim 15 , wherein the spatial audio signal verbally indicates a type of the object.
22 . The method of claim 15 , wherein the spatial audio signal is provided in conjunction with an assistant feature that identifies a certain location or a certain landmark.
23 . The method of claim 15 , wherein the spatial audio signal provides information about a trip taken by the rider in the vehicle.
24 . A vehicle configured to operate in an autonomous driving mode, the vehicle comprising:
a perception system including one or more sensors, the one or more sensors being configured to receive sensor data associated with objects in an external environment of the vehicle and detect a presence of an object in a vicinity of the vehicle;
a driving system including a steering subsystem, an acceleration subsystem and a deceleration subsystem to control driving of the vehicle;
a positioning system configured to identify an actual location of the vehicle and estimate a heading of the vehicle; and
a control system including one or more processors, the control system operatively coupled to the driving system, the perception system, and the positioning system, the control system configured to:
based on detecting the presence of the object in the vicinity of the vehicle, select a sound
generate, based on the selected sound, a spatial audio signal, the spatial audio signal having a directional component according to the actual location of the vehicle, a relative position of the object in the vicinity of the vehicle, the estimated heading of the vehicle, and dynamic detection of an orientation of a rider's head, and the generating includes dynamically adjust the directional component of the spatial audio signal in response to detected changes in the dynamic detection of the orientation of the rider's head and detected changes of the heading of the vehicle; and
provide the spatial audio signal for presentation to the rider within the vehicle.