Object tracking in a head mounted reference frame in an augmented and/or virtual reality environment

1. A computer-implemented method, comprising:

displaying a virtual environment on a display of a head mounted electronic device, the head mounted electronic device operating in an ambient environment;

detecting an ambient object in the ambient environment;

detecting a first ambient position of the ambient object in the ambient environment;

displaying a virtual rendering of the ambient object at a first virtual position in the display of the virtual environment, the first virtual position corresponding to the detected first ambient position of the ambient object in the ambient environment;

detecting a movement of the head mounted electronic device, from a first position of the head mounted electronic device to a second position of the head mounted electronic device;

updating a position of the ambient object in the ambient environment based on the detected movement of the head mounted electronic device, including:

detecting a movement of the ambient object in the ambient environment based on sensor data received from the ambient object; and

determining a second ambient position of the ambient object in the ambient environment based on the detected movement of the ambient object; and

displaying the virtual rendering of the ambient object at a second virtual position in the display of the virtual environment based on the updated position of the ambient object in the ambient environment.

2. The method of claim 1 , detecting the movement of the head mounted electronic device including detecting a change in at least one of a position or an orientation of the head mounted electronic device, and

updating the position of the ambient object in the ambient environment based on the detected movement of the head mounted electronic device including:

detecting a magnitude of the detected movement of the head mounted electronic device; and

maintaining the virtual rendering of the ambient object at the first virtual position in the virtual display of the virtual environment when a magnitude of the detected movement of the head mounted electronic device is less than a previously defined threshold.

3. The method of claim 1 , detecting the movement of the head mounted electronic device including:

detecting a change in at least one of a position or an orientation of the head mounted electronic device, and

updating the position of the ambient object in the ambient environment based on the detected movement of the head mounted electronic device including:

detecting a magnitude of the detected movement of the head mounted electronic device; and

updating the ambient position of the ambient object to a second ambient position, so that the second ambient position of the ambient object relative to the second position of the head mounted electronic device is the same as the first ambient position of the ambient object relative to first position of the head mounted electronic device.

4. The method of claim 1 , receiving the sensor data from the ambient object including receiving sensor data from at least one of a gyroscope or an accelerometer of the ambient object.

5. The method of claim 1 , detecting the movement of the head mounted electronic device, and updating the position of the ambient object in the ambient environment based on the detected movement of the head mounted electronic device including:

detecting a position of the head mounted electronic device with respect to a fixed coordinate system in the ambient environment;

detecting a position of the ambient object with respect to the fixed coordinate system in the ambient environment; and

translating the detected position of the ambient object with respect to the fixed coordinate system in the ambient environment into a translated position of the ambient object with respect to a coordinate system of the head mounted electronic device.

6. The method of claim 5 , updating the position of the ambient object in the ambient environment based on the detected movement of the head mounted electronic device also including:

detecting a magnitude of the detected movement of the head mounted electronic device; and

updating the ambient position of the ambient object to a second ambient position, so that the second ambient position of the ambient object relative to the second position of the head mounted electronic device is the same as the first ambient position of the ambient object relative to first position of the head mounted electronic device.

7. The method of claim 1 , updating the position of the ambient object in the ambient environment based on the detected movement of the head mounted electronic device also including:

receiving sensor data from the head mounted electronic device;

synthesizing the sensor data received from the head mounted electronic device with the sensor data received from the ambient object;

detecting a position of the ambient object with respect to the head mounted electronic device based on the synthesized sensor data; and

repeating the receiving, synthesizing and detecting to update the position of the ambient object with respect to the head mounted electronic device substantially in real time.

8. The method of claim 7 , wherein the ambient object is a handheld electronic device that is operably coupled to the head mounted electronic device, the method further comprising:

synchronizing a time clock of the handheld electronic device with a time clock of the head mounted electronic device, and

synthesizing the sensor data received from the head mounted electronic device with the sensor data received from the ambient object including synthesizing time-stamped sensor data received from the head mounted electronic device with time-stamped sensor data received from the handheld electronic device to update a position of the handheld electronic device with respect to the head mounted electronic device at corresponding points in time.

9. The method of claim 6 , displaying the virtual rendering of the ambient object at the second virtual position in the display of the virtual environment based on the updated position of the ambient object in the ambient environment including:

displaying the virtual rendering of the ambient object at the second virtual position in the virtual display of the virtual environment when the magnitude of the detected movement of the head mounted electronic device is greater than or equal to a previously defined threshold, the second virtual position in the virtual display corresponding to the second ambient position in the ambient environment.

10. The method of claim 6 , displaying the virtual rendering of the ambient object at the second virtual position in the display of the virtual environment based on the updated position of the ambient object in the ambient environment including:

maintaining the virtual rendering of the ambient object at the first virtual position in the virtual display of the virtual environment when the magnitude of the detected movement of the head mounted electronic device is less than a previously defined threshold.

11. A computer program product embodied on a non-transitory computer readable medium, the non-transitory computer readable medium having stored thereon a sequence of instructions which, when executed by a processor, causes the processor to execute a method, the method comprising:

displaying a virtual environment on a display of a head mounted electronic device operating in an ambient environment;

detecting a handheld electronic device in the ambient environment, the handheld electronic device operating in the ambient environment, and the handheld electronic device being operably coupled to the head mounted electronic device;

detecting a first ambient position of the handheld electronic device in the ambient environment;

displaying a virtual rendering of the handheld electronic device at a first virtual position in the display of the virtual environment, the first virtual position corresponding to the detected first ambient position of the handheld electronic device in the ambient environment;

detecting a movement of the head mounted electronic device, from a first position of the head mounted electronic device to a second position of the head mounted electronic device;

updating a position of the handheld electronic device in the ambient environment based on the detected movement of the head mounted electronic device, including:

detecting a position of the head mounted electronic device with respect to a fixed coordinate system in the ambient environment;

detecting a position of the handheld electronic device with respect to the fixed coordinate system in the ambient environment; and

translating the detected position of the handheld electronic device with respect to the fixed coordinate system in the ambient environment into a translated position of the handheld electronic device with respect to a coordinate system of the head mounted electronic device; and

displaying the virtual rendering of the handheld electronic device at a second virtual position in the display of the virtual environment based on the updated position of the handheld electronic device in the ambient environment.

12. The computer program product of claim 11 , updating the position of the handheld electronic device in the ambient environment based on the detected movement of the head mounted electronic device also including:

detecting a magnitude of the detected movement of the head mounted electronic device; and

updating the ambient position of the handheld electronic device to a second ambient position, so that a second ambient position of the handheld electronic device relative to the second position of the head mounted electronic device is the same as the first ambient position of the handheld electronic device relative to first position of the head mounted electronic device.

13. The computer program product of claim 12 , detecting the handheld electronic device in the ambient environment including receiving sensor data from the handheld electronic device, the sensor data including data from at least one of a gyroscope or an accelerometer of the handheld electronic device, and

updating the position of the handheld electronic device in the ambient environment based on the detected movement of the head mounted electronic device also including:

detecting a movement of the handheld electronic device in the ambient environment based on the sensor data received from the handheld electronic device; and

determining the second ambient position of the handheld electronic device in the ambient environment based on the detected movement of the handheld electronic device.

14. The computer program product of claim 13 , updating the position of the handheld electronic device in the ambient environment based on the detected movement of the head mounted electronic device also including:

receiving sensor data from the head mounted electronic device;

synthesizing the sensor data received from the head mounted electronic device with the sensor data received from the handheld electronic device;

detecting a position of the handheld electronic device with respect to the head mounted electronic device based on the synthesized sensor data; and

repeating the receiving, synthesizing and detecting to update the position of the handheld electronic device with respect to the head mounted electronic device substantially in real time.

15. The computer program product of claim 14 , the method further comprising:

synchronizing a time clock of the handheld electronic device with a time clock of the head mounted electronic device, and

synthesizing the sensor data received from the head mounted electronic device with the sensor data received from the ambient object including:

synthesizing time-stamped sensor data received from the head mounted electronic device with time-stamped sensor data received from the handheld electronic device to update a position of the handheld electronic device with respect to the head mounted electronic device at corresponding points in time.

16. The computer program product of claim 11 , displaying the virtual rendering of the handheld electronic device at the second virtual position in the display of the virtual environment based on the updated position of the handheld electronic device in the ambient environment including:

displaying the virtual rendering of the handheld electronic device at the second virtual position in the virtual display of the virtual environment when a magnitude of the detected movement of the head mounted electronic device is greater than or equal to a previously defined threshold, the second virtual position in the virtual display corresponding to the second ambient position in the ambient environment.

17. The computer program product of claim 11 , displaying the virtual rendering of the handheld electronic device at a second virtual position in the display of the virtual environment based on the updated position of the handheld electronic device in the ambient environment including:

maintaining the virtual rendering of the handheld electronic device at the first virtual position in the virtual display of the virtual environment when a magnitude of the detected movement of the head mounted electronic device is less than a previously defined threshold.

18. A computing device, including:

a memory storing executable instructions; and

a processor configured to execute the instructions, to cause the computing device to:

display, on a display of a head mounted electronic device operating in an ambient environment, a virtual environment;

detect a handheld electronic device in the ambient environment, the handheld electronic device operating in the ambient environment, and the handheld electronic device being operably coupled to the head mounted electronic device;

detect a first ambient position of the handheld electronic device in the ambient environment;

display a virtual rendering of the handheld electronic device at a first virtual position in the display of the virtual environment, the first virtual position corresponding to the detected first ambient position of the handheld electronic device in the ambient environment;

detect a movement of the head mounted electronic device;

update a position of the handheld electronic device in the ambient environment based on the detected movement of the head mounted electronic device and sensor data received from the handheld electronic device and the head mounted electronic device;

display the virtual rendering of the handheld electronic device at the first virtual position in the display of the virtual environment when a magnitude of the detected movement is less than a previously defined threshold; and

display the virtual rendering of the handheld electronic device at a second virtual position in the display of the virtual environment when the magnitude of the detected movement is greater than or equal to the previously defined threshold.

19. A computer-implemented method, comprising:

displaying a virtual environment on a display of a head mounted electronic device, the head mounted electronic device operating in an ambient environment;

detecting an ambient object in the ambient environment;

detecting a first ambient position of the ambient object in the ambient environment;

displaying a virtual rendering of the ambient object at a first virtual position in the display of the virtual environment, the first virtual position corresponding to the detected first ambient position of the ambient object in the ambient environment;

detecting a movement of the head mounted electronic device, from a first position of the head mounted electronic device to a second position of the head mounted electronic device, including detecting a magnitude of the detected movement of the head mounted electronic device;

updating a position of the ambient object in the ambient environment based on the detected movement of the head mounted electronic device; and

displaying the virtual rendering of the ambient object in the display of the virtual environment based on the updated position of the ambient object in the ambient environment, including:

maintaining the virtual rendering of the ambient object at the first virtual position in the virtual display of the virtual environment when a magnitude of the detected movement of the head mounted electronic device is less than a previously defined threshold; and

displaying the virtual rendering of the ambient object at a second virtual position in the display of the virtual environment when the magnitude of the detected movement of the head mounted electronic device is greater than or equal to the previously defined threshold.

20. A computer-implemented method, comprising:

displaying a virtual environment on a display of a head mounted electronic device, the head mounted electronic device operating in an ambient environment;

detecting an ambient object in the ambient environment;

detecting a first ambient position of the ambient object in the ambient environment;

displaying a virtual rendering of the ambient object at a first virtual position in the display of the virtual environment, the first virtual position corresponding to the detected first ambient position of the ambient object in the ambient environment;

detecting a movement of the head mounted electronic device, from a first position of the head mounted electronic device to a second position of the head mounted electronic device;

updating a position of the ambient object in the ambient environment based on the detected movement of the head mounted electronic device, including:

detecting a position of the head mounted electronic device with respect to a fixed coordinate system in the ambient environment;

detecting a position of the ambient object with respect to the fixed coordinate system in the ambient environment;

translating the detected position of the ambient object with respect to the fixed coordinate system in the ambient environment into a translated position of the ambient object with respect to a coordinate system of the head mounted electronic device; and

displaying the virtual rendering of the ambient object at a second virtual position in the display of the virtual environment based on the updated position of the ambient object in the ambient environment.

21. A computer-implemented method, comprising:

displaying a virtual environment on a display of a head mounted electronic device, the head mounted electronic device operating in an ambient environment;

detecting an ambient object in the ambient environment;

detecting a first ambient position of the ambient object in the ambient environment;

displaying a virtual rendering of the ambient object at a first virtual position in the display of the virtual environment, the first virtual position corresponding to the detected first ambient position of the ambient object in the ambient environment;

detecting a movement of the head mounted electronic device, from a first position of the head mounted electronic device to a second position of the head mounted electronic device;

updating a position of the ambient object in the ambient environment based on the detected movement of the head mounted electronic device, including:

receiving sensor data from the head mounted electronic device;

synthesizing the sensor data received from the head mounted electronic device with sensor data received from the ambient object;

detecting a position of the ambient object with respect to the head mounted electronic device based on the synthesized sensor data; and

repeating the receiving, synthesizing and detecting to update the position of the ambient object with respect to the head mounted electronic device substantially in real time; and

displaying the virtual rendering of the ambient object at a second virtual position in the display of the virtual environment based on the updated position of the ambient object in the ambient environment.