Augmenting a physical object with virtual components
Systems and methods are presented for immersive and simultaneous animation in a mixed reality environment. Techniques disclosed represent a physical object, present at a scene, in a 3D space of a virtual environment associated with the scene. A virtual element is posed relative to the representation of the physical object in the virtual environment. The virtual element is displayed to users from a perspective of each user in the virtual environment. Responsive to an interaction of one user with the virtual element, an edit command is generated and the pose of the virtual element is adjusted in the virtual environment according to the edit command. The display of the virtual element to the users is then updated according to the adjusted pose. When simultaneous and conflicting edit commands are generated by collaborating users, policies to reconcile the conflicting edit commands are disclosed.
1. A method comprising:
representing a physical object, present at a physical scene, using a three-dimensional (3D) model in a virtual environment associated with the physical scene;
providing a pose for a virtual element in the virtual environment at a location relative to the 3D model of the physical object;
displaying the virtual element to a first user from a perspective of the first user in the virtual environment;
generating a first edit command derived from data received from a controller operated by the first user;
adjusting the pose of the virtual element in the virtual environment according to the first edit command;
updating the displaying of the virtual element to the first user according to the adjusted pose;
representing another physical object, present at the physical scene, using another 3D model in the virtual environment;
detecting a distance, in the virtual environment, between the another 3D model of the another physical object and the virtual element; and
alerting when the detected distance is below a predetermined value.
2. The method of claim 1 , further comprising:
capturing, by a video camera, a video of the physical scene including the physical object, from the perspective of the first user, wherein the displaying of the virtual element to the first user comprises overlaying the virtual element on the captured video of the physical scene.
3. The method of claim 2 , wherein the 3D model of the physical object is constructed based on the captured video.
4. The method of claim 1 , wherein the 3D model of the physical object is constructed based on data measured by one or more sensors associated with the physical object.
5. A method comprising:
representing a physical object, present at a physical scene, using a three-dimensional (3D) model in a virtual environment associated with the physical scene;
providing a pose for a virtual element in the virtual environment at a location relative to the 3D model of the physical object;
displaying the virtual element to a first user from a perspective of the first user in the virtual environment;
generating a first edit command derived from data received from a controller operated by the first user;
adjusting the pose of the virtual element in the virtual environment according to the first edit command;
updating the displaying of the virtual element to the first user according to the adjusted pose;
displaying the virtual element to a second user from a perspective of the second user in the virtual environment;
generating a second edit command derived from second data received from a second controller operated by the second user;
determining a conflict between the first edit command and the second edit command;
reconciling, in response to determining the conflict, the first edit command and the second edit command into a third edit command;
further adjusting the adjusted pose of the virtual element in the virtual environment according to the third edit command; and
further updating the displaying of the virtual element to the first user and to the second user according to the further adjusted pose.
6. The method of claim 5 , wherein the first user and the second user are located remotely from each other and wherein the virtual environment is further associated with a physical scene at which the second user is located.
7. The method of claim 5 , further comprising:
displaying a graphical representation of the first user to the second user from the perspective of the second user in the virtual environment.
8. The method of claim 5 , further comprising:
displaying a graphical representation of the 3D model of the physical object to the second user from the perspective of the second user in the virtual environment.
9. The method of claim 5 , further comprising:
capturing, by a video camera, a video of the physical scene including the physical object, from the perspective of the first user, wherein the displaying of the virtual element to the first user comprises overlaying the virtual element on the captured video of the physical scene, wherein the 3D model of the physical object is constructed based on the captured video.
10. The method of claim 5 , wherein the 3D model of the physical object is constructed based on data measured by one or more sensors associated with the physical object.
11. An apparatus comprising:
at least one processor; and
a memory storing instructions that, when executed by the at least one processor, cause the apparatus to:
represent a physical object, present at a physical scene, using a three-dimensional (3D) model in a virtual environment associated with the physical scene;
provide a pose for a virtual element in the virtual environment at a location relative to the 3D model of the physical object;
display the virtual element to a first user from a perspective of the first user in the virtual environment;
generate a first edit command derived from data received from a controller operated by the first user;
adjust the pose of the virtual element in the virtual environment according to the first edit command;
update the displaying of the virtual element to the first user according to the adjusted pose;
represent another physical object, present at the physical scene, using another 3D model in the virtual environment;
detect a distance, in the virtual environment, between the another 3D model of the another physical object and the virtual element; and
alert when the detected distance is below a predetermined value.
12. The apparatus of claim 11 , wherein the memory storing instructions that, when executed by the at least one processor, further cause the apparatus to:
capture, by a video camera, a video of the physical scene including the physical object, from the perspective of the first user, wherein the display of the virtual element to the first user comprises overlaying the virtual element on the captured video of the physical scene.
13. The apparatus of claim 12 , wherein the 3D model of the physical object is constructed based on the captured video.
14. The apparatus of claim 11 , wherein the 3D model of the physical object is constructed based on data measured by one or more sensors associated with the physical object.
15. An apparatus comprising:
at least one processor; and
a memory storing instructions that, when executed by the at least one processor, cause the apparatus to:
represent a physical object, present at a physical scene, using a three-dimensional (3D) model in a virtual environment associated with the physical scene;
provide a pose for a virtual element in the virtual environment at a location relative to the 3D model of the physical object;
display the virtual element to a first user from a perspective of the first user in the virtual environment;
generate a first edit command derived from data received from a controller operated by the first user;
adjust the pose of the virtual element in the virtual environment according to the first edit command;
update the displaying of the virtual element to the first user according to the adjusted pose;
display the virtual element to a second user from a perspective of the second user in the virtual environment;
generate a second edit command derived from second data received from a second controller operated by the second user;
determine a conflict between the first edit command and the second edit command;
reconcile, in response to determining the conflict, the first edit command and the second edit command into a third edit command;
further adjust the adjusted pose of the virtual element in the virtual environment according to the third edit command; and
further update the displaying of the virtual element to the first user and to the second user according to the further adjusted pose.
16. The apparatus of claim 15 , wherein the first user and the second user are located remotely from each other and wherein the virtual environment is further associated with a physical scene at which the second user is located.
17. The apparatus of claim 15 , wherein the memory storing instructions that, when executed by the at least one processor, further cause the apparatus to:
display a graphical representation of the first user to the second user from the perspective of the second user in the virtual environment.
18. The apparatus of claim 15 , wherein the memory storing instructions that, when executed by the at least one processor, further cause the apparatus to:
display a graphical representation of the 3D model of the physical object to the second user from the perspective of the second user in the virtual environment.
19. The apparatus of claim 15 , wherein the memory storing instructions that, when executed by the at least one processor, further cause the apparatus to:
capture, by a video camera, a video of the physical scene including the physical object, from the perspective of the first user, wherein the displaying of the virtual element to the first user comprises overlaying the virtual element on the captured video of the physical scene, wherein the 3D model of the physical object is constructed based on the captured video.
20. The apparatus of claim 15 , wherein the 3D model of the physical object is constructed based on data measured by one or more sensors associated with the physical object.