System and method for aligning virtual objects on peripheral devices in low-cost augmented reality/virtual reality slip-in systems

1. A system, comprising:

an electronic peripheral for an Augmented Reality (AR) or Virtual Reality (VR) application, the electronic peripheral comprising a first inertial measurement unit (“IMU”), the first IMU comprising:

a first magnetometer;

a first gyroscope; and

a first accelerometer;

a head mounted display, comprising a second IMU, the second IMU comprising:

a second magnetometer;

a second gyroscope; and

a second accelerometer;

a mobile device, comprising a third IMU, the third IMU comprising:

a third magnetometer

a third gyroscope; and

a third accelerometer;

a processor; and

a memory storing a program, which, when executed on the processor, performs an operation, the operation comprising:

receiving a first estimated attitude for the electronic peripheral, the first estimated attitude generated using data from the first IMU;

determining a second estimated attitude for the head mounted display, comprising:

determining an initial second estimated attitude for the head mounted display, at the mobile device, using data from the third gyroscope, and the third accelerometer, in the mobile device; and

after determining the initial second estimated attitude, modifying the determined initial second estimated attitude using data from the second magnetometer in the head mounted display and not from the third magnetometer; and

predicting a future attitude for the head mounted display, for a future time, based on the modified second estimated attitude for the head mounted display determined using data from the second magnetometer and not from the third magnetometer;

determining a first orientation of a virtual object for display on the head mounted display based on the estimated first and second attitudes, such that the virtual object is aligned with the electronic peripheral when the virtual object is displayed to a user; and

displaying the virtual object on the head mounted display using the determined first orientation;

correcting the predicted future attitude for the head mounted display based on measured data from the second IMU;

determining a second orientation of the virtual object for display on the head mounted display based on the corrected predicted future attitude for the head mounted display; and

displaying the virtual object on the head mounted display using the determined second orientation.

2. The system of claim 1 , wherein the first estimated attitude for the electronic peripheral is generated by the electronic peripheral.

3. The system of claim 1 , wherein the mobile device comprises the processor and the memory storing the program.

4. The system of claim 3 , wherein the second estimated attitude is determined by the head mounted display.

5. A computer implemented method of aligning a virtual object with a physical object in an AR or VR application, comprising:

receiving a first estimated attitude for an electronic peripheral for an AR or VR application, the electronic peripheral comprising a first IMU, the first IMU comprising a first magnetometer, a first gyroscope, and a first accelerometer, wherein the first estimated attitude is generated using data from the first IMU;

determining a second estimated attitude for a head mounted display, the head mounted display comprising a second IMU, the second IMU comprising a second magnetometer, a second gyroscope, and a second accelerometer, wherein the determining the second estimated attitude comprises:

determining, at a mobile device, an initial second estimated attitude for the head mounted display using data from a third gyroscope, and a third accelerometer, in the mobile device, wherein the mobile device further comprises a third magnetometer;

after determining the initial second estimated attitude, modifying the determined initial second estimated attitude using data from the second magnetometer in the head mounted display and not from the third magnetometer; and

predicting a future attitude for the head mounted display, for a future time, based on the modified second estimated attitude for the head mounted display determined using data from the second magnetometer display and not from the third magnetometer;

determining a first orientation of a virtual object for display on the head mounted display based on the estimated first and second attitudes, such that the virtual object is aligned with the electronic peripheral when the virtual object is displayed to a user;

displaying the virtual object on the head mounted display using the determined first orientation;

correcting the predicted future attitude for the head mounted display based on measured data from the second IMU;

determining a second orientation of the virtual object for display on the head mounted display based on the corrected predicted future attitude for the head mounted display; and

displaying the virtual object on the head mounted display using the determined second orientation.

6. The method of claim 5 , wherein the first estimated attitude for the electronic peripheral is generated by the electronic peripheral.

7. The method of claim 6 , wherein determining the first orientation of the virtual object further comprises comparing the first estimated attitude for the electronic peripheral and the second estimated attitude for the head mounted display with a plurality of fixed vectors, comprising a first vector relating to gravity and a second vector relating to magnetic north.

8. The method of claim 5 , wherein the second estimated attitude is determined by the head mounted display.

9. A computer program product for aligning a virtual object with a physical object in an AR or VR application, the computer program product comprising:

a non-transitory computer-readable storage medium having computer-readable program code embodied therewith, wherein the code, when executed by a processor, performs an operation, the operation comprising:

receiving a first estimated attitude for an electronic peripheral for an AR or VR application, the electronic peripheral comprising a first IMU, the first IMU comprising a first magnetometer, a first gyroscope, and a first accelerometer, wherein the first estimated attitude is generated using data from the first IMU;

determining a second estimated attitude for a head mounted display, the head mounted display comprising a second IMU, the second IMU comprising a second magnetometer, a second gyroscope, and a second accelerometer, wherein the determining the second estimated attitude comprises:

determining, at a mobile device, an initial second estimated attitude for the head mounted display using data from a third gyroscope, and a third accelerometer, in the mobile device, wherein the mobile device further comprises a third magnetometer;

after determining the initial second estimated attitude, modifying the determined initial second estimated attitude using data from the second magnetometer in the head mounted display and not from the third magnetometer; and

predicting a future attitude for the head mounted display, for a future time, based on the modified second estimated attitude for the head mounted display determined using data from the second magnetometer and not from the third magnetometer;

determining a first orientation of a virtual object for display on the head mounted display based on the estimated first and second attitudes, such that the virtual object is aligned with the electronic peripheral when the virtual object is displayed to a user;

displaying the virtual object on the head mounted display using the determined first orientation;

correcting the predicted future attitude for the head mounted display based on measured data from the second IMU;

determining a second orientation of the virtual object for display on the head mounted display based on the corrected predicted future attitude for the head mounted display; and

displaying the virtual object on the head mounted display using the determined second orientation.

10. The computer program product of claim 9 , wherein the first estimated attitude for the electronic peripheral is generated by the electronic peripheral.

11. The computer program product of claim 9 , wherein determining the first orientation of the virtual object further comprises comparing the first estimated attitude for the electronic peripheral and the second estimated attitude for the head mounted display with a plurality of fixed vectors, comprising a first vector relating to gravity and a second vector relating to magnetic north.

12. The computer program product of claim 9 , wherein the second estimated attitude is determined by the head mounted display.

13. The system of claim 1 , wherein the predicting the future attitude for the head mounted display is further based on a third estimated attitude for the head mounted display determined at a different time than the second estimated attitude for the head mounted display.

14. The method of claim 5 , wherein the predicting the future attitude for the head mounted display is further based on a third estimated attitude for the head mounted display determined at a different time than the second estimated attitude for the head mounted display.

15. The system of claim 1 , wherein determining the first orientation of the virtual object further comprises comparing the first estimated attitude for the electronic peripheral and the second estimated attitude for the head mounted display with a plurality of fixed vectors, comprising a first vector relating to gravity and a second vector relating to magnetic north.