Location-based virtual element modality in three-dimensional content
Various implementations disclosed herein include devices, systems, and methods that enable a device to provide a view of virtual elements and a physical environment where the presentation of the virtual elements is based on positioning relative to the physical environment. In one example, a device is configured to detect a change in positioning of a virtual element, for example, when a virtual element is added, moved, or the physical environment around the virtual element is changed. The location of the virtual element in the physical environment is used to detect an attribute of the physical environment upon which the presentation of the virtual element depends. Thus, the device is further configured to detect an attribute (e.g., surface, table, mid-air, etc.) of the physical environment based on the placement of the virtual element and present the virtual element based on the detected attribute.
1. A method comprising:
at a device with a processor and a computer readable medium:
providing a view comprising a virtual element and a physical environment, wherein the virtual element is a first type of modality and is at a first location in the view relative to the physical environment;
detecting a change in the view, the change comprising placement of the virtual element at a second location in the view; and
in response to detecting the change in the view based on the placement of the virtual element at the second location:
detecting an attribute of the physical environment at the second location of the virtual element in the view;
selecting a second type of modality of the virtual element based on the detected attribute, wherein the first type of modality comprises a first set of interactive features and the second type of modality comprises a second set of interactive features;
determining, based on the selected second type of modality and the second location of the virtual element in the view, whether to update the virtual element;
in accordance with determining to update the virtual element, updating the virtual element based on the selected type of modality, wherein updating the virtual element comprises:
changing a size, shape, design, or color attribute of the virtual element, and
changing modality-specific functions of the virtual element from the first set of interactive features to the second set of interactive features; and
presenting an updated view of the updated virtual element and the physical environment.
2. The method of claim 1 , wherein providing the view comprises:
obtaining, using an image sensor, an image of the physical environment;
generating a representation of the physical environment using the obtained image; and
presenting the virtual element with the representation of the physical environment.
3. The method of claim 1 , wherein providing the view comprises:
identifying, using an image sensor, a feature in the physical environment; and
presenting the virtual element on an optical see-through display responsive to identifying the feature.
4. The method of claim 1 , wherein selecting the second type of modality of the virtual element based on the detected attribute comprises:
in accordance with determining that the attribute is a first attribute, updating the virtual element to have a first visual appearance when updating the virtual element; and
in accordance with determining that the attribute is a second attribute, updating the virtual element to have a second visual appearance when updating the virtual element, wherein the first visual appearance is different from the second visual appearance.
5. The method of claim 1 , wherein selecting the second type of modality of the virtual element based on the detected attribute comprises:
in accordance with determining that the attribute is a first attribute, updating the virtual element to be updated to have a first functionality when updating the virtual element; and
in accordance with determining that the attribute is a second attribute, updating the virtual element to be updated to have a second functionality when updating the virtual element, wherein the first functionality is different from the second functionality.
6. The method of claim 1 , wherein detecting the change in the view comprises receiving user input representing a change in the location of the virtual element.
7. The method of claim 1 , wherein selecting the second type of modality of the virtual element based on the detected attribute comprises determining that the virtual element is on a planar surface.
8. The method of claim 1 , wherein selecting the second type of modality of the virtual element based on the detected attribute comprises determining that the virtual element is on a horizontal planar surface.
9. The method of claim 1 , wherein selecting the second type of modality of the virtual element based on the detected attribute comprises determining that the virtual element is on a vertical planar surface.
10. The method of claim 1 , wherein detecting the attribute comprises detecting whether the second location is a planar surface and wherein selecting the second type of modality of the virtual element based on the detected attribute comprises determining that the planar surface is a wall.
11. The method of claim 1 , wherein detecting the attribute comprises detecting whether the second location is a planar surface and wherein selecting the second type of modality of the virtual element based on the detected attribute comprises determining that the planar surface is a table.
12. The method of claim 1 , wherein detecting the attribute comprises detecting whether the second location is a planar surface and wherein selecting the second type of modality of the virtual element based on the detected attribute comprises determining that the planar surface is a floor.
13. The method of claim 1 , wherein detecting the attribute comprises detecting whether the second location is a planar surface and wherein selecting the second type of modality of the virtual element based on the detected attribute comprises determining that the planar surface is a ramp.
14. The method of claim 1 , wherein detecting the attribute comprises detecting whether the second location is a planar surface and wherein selecting the second type of modality of the virtual element based on the detected attribute comprises determining that the planar surface is a stairway.
15. The method of claim 1 , wherein detecting the attribute comprises detecting whether the second location is a planar surface and wherein selecting the second type of modality of the virtual element based on the detected attribute comprises determining that the planar surface is a ceiling.
16. The method of claim 1 , wherein detecting the attribute comprises:
identifying a classification of a portion of the physical environment using a classifier; and
identifying an amount of the portion available upon which the virtual element can be positioned.
17. The method of claim 1 , wherein presenting an updated view of the updated virtual element and the physical environment comprises:
in accordance with determining that the attribute is of a first attribute, displaying the updated virtual element at the second location; and
in accordance with determining that the attribute is of a second attribute, displaying the updated virtual element away from the second location, wherein the first attribute is different from the second attribute.
18. The method of claim 1 , wherein the modality of the virtual element is based on a function of the virtual element.
19. A non-transitory computer-readable storage medium storing program instructions that are computer-executable to perform operations comprising:
providing a view, at a device, the view comprising a virtual element and a physical environment, wherein the virtual element is a first type of modality and is at a first location in the view relative to the physical environment;
detecting a change in the view, the change comprising placement of the virtual element at a second location in the view; and
in response to detecting the change in the view based on the placement of the virtual element at the second location:
detecting an attribute of the physical environment at the second location of the virtual element in the view;
selecting a second type of modality of the virtual element based on the detected attribute, wherein the first type of modality comprises a first set of interactive features and the second type of modality comprises a second set of interactive features;
determining, based on the selected second type of modality and the second location of the virtual element in the view, whether to update the virtual element;
in accordance with determining to update the virtual element, updating the virtual element based on the selected type of modality, wherein updating the virtual element comprises:
changing a size, shape, design, or color attribute of the virtual element, and
changing modality-specific functions of the virtual element from the first set of interactive features to the second set of interactive features; and
presenting an updated view of the updated virtual element and the physical environment.
20. The non-transitory computer-readable storage medium of claim 19 , wherein detecting the attribute comprises detecting whether the second location is a planar surface and wherein selecting the second type of modality of the virtual element based on the detected attribute comprises determining that the planar surface is a wall, table, floor, ramp, stairway, or ceiling.
21. The non-transitory computer-readable storage medium of claim 19 , wherein detecting the attribute comprises:
identifying a classification of a portion of the physical environment using a classifier; and
identifying an amount of the portion available upon which the virtual element can be positioned.
22. A system comprising:
an electronic device with a display and an image sensor;
a processor; and
a computer-readable storage medium comprising instructions that upon execution by the processor cause the system to perform operations, the operations comprising:
providing a view comprising a virtual element and a physical environment wherein the virtual element is a first type of modality and is at a first location in the view relative to the physical environment;
detecting a change in the view, the change comprising placement of the virtual element at a second location in the view; and
in response to detecting the change in the view based on the placement of the virtual element at the second location:
detecting an attribute of the physical environment at the second location of the virtual element in the view;
selecting a second type of modality of the virtual element based on the detected attribute, wherein the first type of modality comprises a first set of interactive features and the second type of modality comprises a second set of interactive features;
determining, based on the selected second type of modality and the second location of the virtual element in the view, whether to update the virtual element;
in accordance with determining to update the virtual element, updating the virtual element based on the selected type of modality, wherein updating the virtual element comprises:
changing a size, shape, design, or color attribute of the virtual element, and
changing modality-specific functions of the virtual element from the first set of interactive features to the second set of interactive features; and
presenting an updated view of the updated virtual element and the physical environment.
23. The system of claim 22 , wherein detecting the attribute comprises detecting whether the second location is a planar surface and wherein selecting the second type of modality of the virtual element based on the detected attribute comprises determining that the planar surface is a wall, table, floor, ramp, stairway, or ceiling.
24. The system of claim 22 , wherein detecting the attribute comprises:
identifying a classification of a portion of the physical environment using a classifier; and
identifying an amount of the portion available upon which the virtual element can be positioned.