Representing real-world objects with a virtual reality environment
An image processing system enables a user wearing a head-mounted display to experience a virtual environment combined with a representation of a real-world object. The image processing system receives a captured scene of a real-world environment that includes a target object. The image processing system identifies the target object in the captured scene and generates a representation of the target object. In some cases, the image processing system may include a graphical overlay with the representation of the target objet. The image processing system can generate a combined scene that includes the target object and the virtual environment. The combined scene is presented to the user, thereby allowing the user to interact with the real-world target object (or a representation thereof) in combination with the virtual environment.
1. A computer-implemented method comprising:
receiving, by a virtual reality system, image data for an image frame of a real-world environment captured by an imaging sensor from a perspective of a head-mounted display worn by a wearer, the imaging sensor configured to capture a view of the real-world environment that the wearer would perceive if the wearer was not wearing the head-mounted display;
identifying, by the virtual reality system, a first set of pixels of the image data representing a target object in the received image frame, the target object included in the real-world environment and identified using object recognition by determining edges that contain the first set of pixels of the image data;
generating, by the virtual reality system, a scene of a virtual environment based on a set of virtual objects;
generating, by the virtual reality system, a combined scene that includes the scene of the virtual environment and the first set of pixels of the image data representing the target object without modification of the image data; and
presenting, by the virtual reality system, the combined scene on the head-mounted display.
2. The method of claim 1 , wherein presenting the combined scene on the head mounted display comprises:
rendering the first set of pixels of the image data representing the target object at a set of coordinates and virtual content of the virtual environment rendered at all other locations of the scene.
3. The method of claim 1 , wherein the first set of pixels of the image data representing the target object includes one or more graphical overlays.
4. The method of claim 1 , wherein the target object is associated with at least one of a marker, a tag, a coloring, or a painting, and wherein the first set of pixels of the image data representing the target object is based on the at least one of the marker, the tag, the coloring, or the painting.
5. The method of claim 4 , wherein the target object is associated with a marker that emits a wireless signal, wherein the first set of pixels of the image data representing the target object includes a graphical overlay selected based on the wireless signal.
6. A system comprising:
at least one processor; and
a memory storing instructions that, when executed by the at least one processor, cause the system to perform:
receiving image data for an image frame of a real-world environment captured by an imaging sensor from a perspective of a head-mounted display worn by a wearer, the imaging sensor configured to capture a view of the real-world environment that the wearer would perceive if the wearer was not wearing the head-mounted display;
identifying a first set of pixels of the image data representing a target object in the received image frame, the target object included in the real-world environment and identified using object recognition by determining edges that contain the first set of pixels of the image data;
generating a scene of a virtual environment based on a set of virtual objects;
generating a combined scene that includes the scene of the virtual environment and the first set of pixels of the image data representing the target object without modification of the image data; and
presenting the combined scene on the head-mounted display.
7. The system of claim 6 , wherein presenting the combined scene on the head mounted display comprises:
rendering the first set of pixels of the image data representing the target object at a set of coordinates and virtual content of the virtual environment rendered at all other locations of the scene.
8. A non-transitory computer-readable storage medium including instructions that, when executed by at least one processor of a computing system, cause the computing system to perform a method comprising:
receiving image data for an image frame of a real-world environment captured by an imaging sensor from a perspective of a head-mounted display worn by a wearer, the imaging sensor configured to capture a view of the real-world environment that the wearer would perceive if the wearer was not wearing the head-mounted display;
identifying a first set of pixels of the image data representing a target object in the received image frame, the target object included in the real-world environment and identified using object recognition by determining edges that contain the first set of pixels of the image data;
generating a scene of a virtual environment based on a set of virtual objects;
generating a combined scene that includes the scene of the virtual environment and the first set of pixels of the image data representing the target object without modification of the image data; and
presenting the combined scene on the head-mounted display.
9. The non-transitory computer-readable storage medium of claim 8 , wherein presenting the combined scene on the head mounted display comprises:
rendering the first set of pixels of the image data representing the target object at a set of coordinates and virtual content of the virtual environment rendered at all other locations of the scene.