OLFACTORY PREDICTIONS USING NEURAL NETWORKS
Methods, systems, and apparatus, including computer programs encoded on computer storage media, for generating olfactory predictions using neural networks. One of the methods includes receiving scene data characterizing a scene in an environment; processing the scene data using a representation neural network to generate a representation of the scene; and processing the representation of the scene using a prediction neural network to generate as output an olfactory prediction that characterizes a predicted smell of the scene at a particular observer location.
1 . A method performed by one or more computers, the method comprising:
receiving scene data characterizing a scene in an environment;
processing the scene data using a representation neural network to generate a representation of the scene; and
processing the representation of the scene using a prediction neural network to generate as output an olfactory prediction that characterizes a predicted smell of the scene at a particular observer location.
2 . The method of claim 1 , wherein the input scene data is an image or a video of the environment, and wherein the particular observer location is a location of the camera that captured the image or video in the environment.
3 . The method of claim 1 , further comprising:
providing the olfactory prediction to a hardware device that is configured to generate the predicted smell.
4 . The method of claim 1 , wherein the olfactory prediction is a prediction of a scent at the particular observer location along a plurality of olfactory dimensions.
5 . The method of claim 4 , wherein each olfactory dimension is represented by a basis vector and corresponds to a different known smell.
6 . The method of claim 4 , wherein the olfactory prediction includes a respective score for each of the olfactory dimensions.
7 . The method of claim 1 , wherein the representation of the scene identifies a plurality of portions of the scene data that depict objects, and wherein processing the representation of the scene using a prediction neural network to generate as output an olfactory prediction that characterizes a predicted smell of the scene at a particular observer location comprises:
for each identified object, processing an object representation characterizing the object using the prediction neural network to generate an object olfactory prediction that characterizes a contribution of the corresponding object to the smell of the scene at the particular observer location; and
determining the olfactory prediction from the object olfactory predictions.
8 . The method of claim 7 , wherein determining the olfactory prediction comprises:
obtaining, for each identified object, a distance of the identified object from the particular observer location; and
summing over the olfactory predictions with each olfactory prediction weighted inversely by the distance for the corresponding object.
9 . A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform operations comprising:
receiving scene data characterizing a scene in an environment;
processing the scene data using a representation neural network to generate a representation of the scene; and
processing the representation of the scene using a prediction neural network to generate as output an olfactory prediction that characterizes a predicted smell of the scene at a particular observer location.
10 . The system of claim 9 , wherein the input scene data is an image or a video of the environment, and wherein the particular observer location is a location of the camera that captured the image or video in the environment.
11 . The system of claim 9 , the operations further comprising:
providing the olfactory prediction to a hardware device that is configured to generate the predicted smell.
12 . The system of claim 9 , wherein the olfactory prediction is a prediction of a scent at the particular observer location along a plurality of olfactory dimensions.
13 . The system of claim 12 , wherein each olfactory dimension is represented by a basis vector and corresponds to a different known smell.
14 . The system of claim 12 , wherein the olfactory prediction includes a respective score for each of the olfactory dimensions.
15 . The system of claim 9 , wherein the representation of the scene identifies a plurality of portions of the scene data that depict objects, and wherein processing the representation of the scene using a prediction neural network to generate as output an olfactory prediction that characterizes a predicted smell of the scene at a particular observer location comprises:
for each identified object, processing an object representation characterizing the object using the prediction neural network to generate an object olfactory prediction that characterizes a contribution of the corresponding object to the smell of the scene at the particular observer location; and
determining the olfactory prediction from the object olfactory predictions.
16 . The system of claim 15 , wherein determining the olfactory prediction comprises:
obtaining, for each identified object, a distance of the identified object from the particular observer location; and
summing over the olfactory predictions with each olfactory prediction weighted inversely by the distance for the corresponding object.
17 . One or more non-transitory computer-readable storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:
receiving scene data characterizing a scene in an environment;
processing the scene data using a representation neural network to generate a representation of the scene; and
processing the representation of the scene using a prediction neural network to generate as output an olfactory prediction that characterizes a predicted smell of the scene at a particular observer location.
18 . The computer-readable storage media of claim 17 , wherein the input scene data is an image or a video of the environment, and wherein the particular observer location is a location of the camera that captured the image or video in the environment.
19 . The computer-readable storage media of claim 17 , the operations further comprising:
providing the olfactory prediction to a hardware device that is configured to generate the predicted smell.
20 . The computer-readable storage media of claim 17 , wherein the olfactory prediction is a prediction of a scent at the particular observer location along a plurality of olfactory dimensions.