Localization using audio and visual data
A reference image and recorded sound of an environment of a client device are obtained. The recorded sound may be captured by a microphone of the client device in a period of time after generation of a localization sound by the client device. The location of the client device in the environment may be determined using the reference image and the recorded sound.
1 . A computer-implemented method for localization, the method comprising:
obtaining a reference image of an environment of a client device;
obtaining recorded sound of the environment of the client device, the recorded sound captured by a microphone of the client device in a period of time after generation of a localization sound by the client device; and
determining a location of the client device in the environment using the reference image and the recorded sound, wherein determining the location comprises:
extracting features from the recorded sound and the reference image;
determining candidate poses based on the features extracted from the recorded sound and the reference image; and
selecting between the candidate poses, wherein selecting between the candidate poses comprises:
providing the candidate poses as input to a learned gating function; and
obtaining from the learned gating function an output indicating whether to use a candidate pose based on the recorded sound or a candidate pose based on the reference image.
2 . The method of claim 1 further comprising:
obtaining a second recorded sound, wherein the second recorded sound is a recorded sound of the environment captured by a second microphone in the period of time after generation of the localization sound by a second client device; and
determining a location of the second microphone relative to the location of the client device in the environment using the second recorded sound.
3 . The method of claim 2 further comprising:
obtaining a second reference image of the environment based on a second client device; and
determining a location of the second client device relative to the location of the client device in the environment using the second recorded sound and the second reference image.
4 . The method of claim 1 further comprising:
querying an audio-visual database based on the reference image of the environment and the recorded sound; and
determining the location of the client device based on query of the audio-visual database.
5 . The method of claim 1 further comprising:
receiving, at a server, the reference image and the recorded sound, wherein the determining of the location of the client device in the environment is performed by the server; and
providing, to the client device, the location of the client device.
6 . The method of claim 1 wherein the location of the client device includes a description of an orientation of the client device with respect to the environment.
7 . The method of claim 1 , wherein the determining of the location of the client device in the environment is determined at the client device.
8 . The method of claim 1 , wherein determining candidate poses based on the features extracted from the recorded sound and the reference image further comprises comparing the features extracted from the recorded sound and the reference image to features from an audio-visual database.
9 . A non-transitory computer-readable medium storing computer-executable instructions for localization that, when executed by a computing system, cause the computing system, to perform operations comprising:
obtaining a reference image of an environment of a client device;
obtaining recorded sound of the environment of the client device, the recorded sound captured by a microphone of the client device in a period of time after generation of a localization sound by the client device; and
determining a location of the client device in the environment using the reference image and the recorded sound, wherein determining the location comprises:
extracting features from the recorded sound and the reference image;
determining candidate poses based on the features extracted from the recorded sound and the reference image; and
selecting between the candidate poses, wherein selecting between the candidate poses comprises:
providing the candidate poses as input to a learned gating function; and
obtaining from the learned gating function an output indicating whether to use a candidate pose based on the recorded sound, a candidate pose based on the reference image, or an optimal combination of the candidate poses.
10 . The computer-readable medium of claim 9 wherein the operations further comprise:
obtaining a second recorded sound, wherein the second recorded sound is a recorded sound of the environment captured by a second microphone in the period of time after generation of the localization sound by a second client device; and
determining a location of the second microphone relative to the location of the client device in the environment using the second recorded sound.
11 . The computer-readable medium of claim 10 wherein the operations further comprise:
obtaining a second reference image of the environment based on a second client device;
and determining a location of the second client device relative to the location of the client device in the environment using the second recorded sound and the second reference image.
12 . The computer-readable medium of claim 9 wherein the operations comprise:
querying an audio-visual database based on the reference image of the environment and the recorded sound; and
determining the location of the client device based on query of the audio-visual database.
13 . The computer-readable medium of claim 9 wherein the location of the client device includes a description of an orientation of the client device with respect to the environment.
14 . A computer-implemented method for localization, the method comprising:
obtaining a reference image of an environment of a client device;
obtaining recorded sound of the environment of the client device, the recorded sound captured by a microphone of the client device in a period of time after generation of a localization sound by the client device; and
determining a location of the client device in the environment using the reference image and the recorded sound, wherein determining the location comprises:
extracting features from the recorded sound and the reference image;
determining candidate poses based on the features extracted from the recorded sound and the reference image; and
selecting between the candidate poses, wherein selecting between the candidate poses comprises:
providing the candidate poses as input to a learned gating function; and
obtaining from the learned gating function an output indicating an optimal combination of the candidate poses.
15 . The method of claim 14 further comprising:
obtaining a second recorded sound, wherein the second recorded sound is a recorded sound of the environment captured by a second microphone in the period of time after generation of the localization sound by a second client device; and
determining a location of the second microphone relative to the location of the client device in the environment using the second recorded sound.
16 . The method of claim 15 further comprising:
obtaining a second reference image of the environment based on a second client device; and
determining a location of the second client device relative to the location of the client device in the environment using the second recorded sound and the second reference image.
17 . The method of claim 14 further comprising:
querying an audio-visual database based on the reference image of the environment and the recorded sound; and
determining the location of the client device based on query of the audio-visual database.
18 . The method of claim 14 further comprising:
receiving, at a server, the reference image and the recorded sound, wherein the determining of the location of the client device in the environment is performed by the server; and
providing, to the client device, the location of the client device.
19 . The method of claim 14 wherein the location of the client device includes a description of an orientation of the client device with respect to the environment.
20 . The method of claim 14 , wherein determining candidate poses based on the features extracted from the recorded sound and the reference image further comprises comparing the features extracted from the recorded sound and the reference image to features from an audio-visual database.