Contextualized visual search
Various implementations disclosed herein include devices, systems, and methods that provide a view of a physical environment with a visual effect based on determining whether an object is a searched-for object. For example, an example process may include determining an objective corresponding to identifying a searched-for object in a physical environment including one or more objects. The process may further include obtaining depth-based data based on sensor data captured by one or more sensors in the physical environment. The process may further include determining, based on the determined objective and the depth-based data, whether a first object in the physical environment is the searched-for object. The process may further include, in accordance with determining whether the first object is the searched-for object, providing a view of the physical environment with a visual effect.
1 . A method comprising:
at an electronic device having a processor:
determining an objective corresponding to identifying a searched-for object in a physical environment comprising one or more objects;
obtaining depth-based data based on sensor data captured by one or more sensors in the physical environment;
determining, based on the determined objective and the depth-based data, whether a first object in the physical environment is the searched-for object; and
in accordance with determining that the first object is the searched-for object, providing a view of the physical environment with a visual effect reducing an appearance characteristic of a second object other than the first object, wherein reducing the appearance characteristic comprises at least one of: removing the second object from the view, blurring the second object, changing a translucency of the second object, changing a color of the second object, and rendering the second object as grayed-out, matte, or frosted.
2 . The method of claim 1 , wherein determining the objective corresponding to identifying the searched-for object in the physical environment is based on physiological data obtained via the one or more sensors, and wherein the physiological data comprises:
a direction of a gaze;
a velocity of the gaze;
positional information and velocity information for a left eye and a right eye;
interpupillary distance (IPD) between the right eye and the left eye, and a diameter of the left eye and a diameter of the right eye;
pupillary fixations;
an image of an eye or electrooculography (EOG) data; or
a pose and a movement of a hand or a head.
3 . The method of claim 1 , wherein, in accordance with determining that the first object is the searched-for object, providing the view of the physical environment with the visual effect comprises providing visual effects for a subset of objects of the one or more objects that are within a proximity threshold to the first object.
4 . The method of claim 1 , wherein, in accordance with determining that the first object is the searched-for object, providing the view of the physical environment with the visual effect comprises at least one of:
blurring a third object within the view of the physical environment;
applying a translucent visual effect to a fourth object within the view of the physical environment;
applying a white-out or grey-out visual effect to a fifth object within the view of the physical environment;
applying a matte white visual effect to a sixth object within the view of the physical environment; or
applying a frosted glass visual effect to a seventh object within the view of the physical environment.
5 . The method of claim 1 , wherein, in accordance with determining that the first object is not the searched-for object, providing the view of the physical environment with the visual effect comprises providing an indication that the first object is not the searched-for object.
6 . The method of claim 1 , wherein determining that the first object in the physical environment is the searched-for object is based on an object detection technique that identifies occlusions between a portion of the one or more objects.
7 . The method of claim 1 , wherein the depth-based data comprises a depth image of the physical environment or a 3D representation of the physical environment.
8 . The method of claim 1 , wherein the device is a head-mounted device (HMD), and wherein the view of the physical environment comprises an extended reality (XR) experience.
9 . The method of claim 1 , wherein reducing the appearance characteristic of the second object indicates that the second object is not the searched-for object.
10 . The method of claim 1 , further comprising reducing the appearance characteristic of a portion of the physical environment within a proximity threshold of the first object.
11 . The method of claim 1 , wherein reducing the appearance characteristic comprises removing the second object from the view.
12 . The method of claim 1 , wherein reducing the appearance characteristic comprises blurring the second object.
13 . The method of claim 1 , wherein reducing the appearance characteristic comprises changing a translucency of the second object.
14 . The method of claim 1 , wherein reducing the appearance characteristic comprises changing a color of the second object.
15 . The method of claim 1 , wherein reducing the appearance characteristic comprises altering the second object to appear grayed-out, matte, or frosted.
16 . A device comprising:
a non-transitory computer-readable storage medium; and
one or more processors coupled to the non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium comprises program instructions that, when executed on the one or more processors, cause the one or more processors to perform operations comprising:
determining an objective corresponding to identifying a searched-for object in a physical environment comprising one or more objects;
obtaining depth-based data based on sensor data captured by one or more sensors in the physical environment;
determining, based on the determined objective and the depth-based data, whether a first object in the physical environment is the searched-for object; and
in accordance with determining that the first object is the searched-for object, providing a view of the physical environment with a visual effect reducing an appearance characteristic of a second object other than the first object, wherein reducing the appearance characteristic comprises at least one of: removing the second object from the view, blurring the second object, changing a translucency of the second object, changing a color of the second object, and rendering the second object as grayed-out, matte, or frosted.
17 . The device of claim 16 , wherein determining the objective corresponding to identifying the searched-for object in the physical environment is based on physiological data obtained via the one or more sensors, and wherein the physiological data comprises:
a direction of a gaze;
a velocity of the gaze;
positional information and velocity information for a left eye and a right eye;
interpupillary distance (IPD) between the right eye and the left eye, and a diameter of the left eye and a diameter of the right eye;
pupillary fixations;
an image of an eye or electrooculography (EOG) data; or
a pose and a movement of a hand or a head.
18 . The device of claim 16 , wherein, in accordance with determining that the first object is the searched-for object, providing the view of the physical environment with the visual effect comprises providing visual effects for a subset of objects of the one or more objects that are within a proximity threshold to the first object.
19 . The device of claim 16 , wherein, in accordance with determining that the first object is the searched-for object, providing the view of the physical environment with the visual effect comprises at least one of:
blurring a third object within the view of the physical environment;
applying a translucent visual effect to a fourth object within the view of the physical environment;
applying a white-out or grey-out visual effect to a fifth object within the view of the physical environment;
applying a matte white visual effect to a sixth object within the view of the physical environment; or
applying a frosted glass visual effect to a seventh object within the view of the physical environment.
20 . The device of claim 16 , wherein, in accordance with determining that the first object is not the searched-for object, providing the view of the physical environment with the visual effect comprises providing an indication that the first object is not the searched-for object.
21 . The device of claim 16 , wherein determining whether the first object in the physical environment is the searched-for object is based on an object detection technique that identifies occlusions between a portion of the one or more objects.
22 . The device of claim 16 , wherein the depth-based data comprises a depth image of the physical environment or a 3D representation of the physical environment.
23 . The device of claim 16 , wherein the device is a head-mounted device (HMD), and wherein the view of the physical environment comprises an extended reality (XR) experience.
24 . A non-transitory computer-readable storage medium, storing program instructions executable on a device including one or more processors to perform operations comprising:
determining an objective corresponding to identifying a searched-for object in a physical environment comprising one or more objects;
obtaining depth-based data based on sensor data captured by one or more sensors in the physical environment;
determining, based on the determined objective and the depth-based data, whether a first object in the physical environment is the searched-for object; and
in accordance with determining that the first object is the searched-for object, providing a view of the physical environment with a visual effect reducing an appearance characteristic of a second object other than the first object, wherein reducing the appearance characteristic comprises at least one of: removing the second object from the view, blurring the second object, changing a translucency of the second object, changing a color of the second object, and rendering the second object as grayed-out, matte, or frosted.