System and method for synchronizing tracking points

1. An electronic device for tracking point synchronization comprising:

a frame-based camera;

an event-based camera; and

a processor coupled to the frame-based camera and the event-based camera, the processor configured to:

receive frame-based data captured by the frame-based camera and event-based data captured by the event-based camera;

identify a frame-based tracking point from the frame-based data;

identify an event-based tracking point from the event-based data corresponding to the frame-based tracking point; and

synchronize the frame-based camera and the event-based camera based on the identified frame-based tracking point and the identified event-based tracking point.

2. The electronic device of claim 1 , wherein:

the event-based data includes a plurality of event-based tracking points,

the processor is further configured to:

determine a quality of each event-based tracking point of the plurality of event-based tracking points compared to the frame-based tracking point;

rank the plurality of event-based tracking points;

select the identified event-based tracking point as a highest ranked tracking point of the plurality of ranked event-based tracking points for use in head pose tracking.

3. The electronic device of claim 2 , wherein a rank of the event-based tracking point includes a stability of an output, a clarity of the event-based tracking point, and a lack of neighboring tracking points.

4. The electronic device of claim 1 , wherein:

the frame-based data includes a frame and a corresponding frame time stamp;

the event-based data includes an output and a corresponding output time stamp; and

the output time stamp for the output corresponding to the event-based tracking point is synchronized with the frame time stamp for the frame corresponding to the frame-based tracking point.

5. The electronic device of claim 1 , wherein the processor is further configured to:

determine a frame-based pose from the frame-based data;

determine an event-based pose from the event-based data; and

monitor the frame-based pose and the event-based pose to be within a tolerance.

6. The electronic device of claim 5 , wherein the event-based camera captures a second output before the frame-based camera captures a second frame.

7. The electronic device of claim 6 , wherein the processor is further configured to:

identify a second event-based tracking point from the event-based data corresponding to the second event; and

synchronize the frame-based camera and the event-based camera based on the identified frame-based tracking point and the identified second event-based tracking point.

8. A method for operating an electronic device for tracking point synchronization comprising:

receiving frame-based data captured by a frame-based camera and event-based data captured by an event-based camera;

identifying a frame-based tracking point from the frame-based data;

identifying an event-based tracking point from the event-based data corresponding to the frame-based tracking point; and

synchronizing the frame-based camera and the event-based camera based on the identified frame-based tracking point and the identified event-based tracking point.

9. The method of claim 8 , wherein the event-based data includes a plurality of event-based tracking points, and

further comprising:

determining a quality of each event-based tracking point of the plurality of event-based tracking points compared to the frame-based tracking point;

ranking the plurality of event-based tracking points; and

selecting the identified event-based tracking point as a highest ranked tracking point of the plurality of ranked event-based tracking points for use in head pose tracking.

10. The method of claim 9 , wherein a rank of the event-based tracking point includes a stability of an output, a clarity of the event-based tracking point, and a lack of neighboring tracking points.

11. The method of claim 8 , wherein:

the frame-based data includes a frame and a corresponding frame time stamp;

the event-based data includes an output and a corresponding output time stamp; and

the output time stamp for the output corresponding to the event-based tracking point is synchronized with the frame time stamp for the frame corresponding to the frame-based tracking point.

12. The method of claim 8 , further comprising:

determining a frame-based pose from the frame-based data;

determining an event-based pose from the event-based data; and

monitoring the frame-based pose and the event-based pose to be within a tolerance.

13. The method of claim 12 , wherein the event-based camera captures a second output before the frame-based camera captures a second frame.

14. The method of claim 13 , further comprising:

identifying a second event-based tracking point from the event-based data corresponding to the second output; and

synchronizing the frame-based camera and the event-based camera based on the identified frame-based tracking point and the identified second event-based tracking point.

15. A non-transitory computer readable medium embodying a computer program, the computer program comprising computer readable program code that when executed causes at least one processor to:

receive frame-based data captured by a frame-based camera and event-based data captured by an event-based camera;

identify a frame-based tracking point from the frame-based data;

identify an event-based tracking point from the event-based data corresponding to the frame-based tracking point; and

synchronize the frame-based camera and the event-based camera based on the identified frame-based tracking point and the identified event-based tracking point.

16. The non-transitory computer readable medium of claim 15 , wherein:

the event-based data includes a plurality of event-based tracking points, and

the computer readable program code that when executed further causes at least one processor to:

determine a quality of each event-based tracking point of the plurality of event-based tracking points compared to the frame-based tracking point;

rank the plurality of event-based tracking points; and

select the identified event-based tracking point as a highest ranked tracking point of the plurality of ranked event-based tracking points for use in head pose tracking.

17. The non-transitory computer readable medium of claim 16 , wherein a rank of the event-based tracking point includes a stability of an output, a clarity of the event-based tracking point, and a lack of neighboring tracking points.

18. The non-transitory computer readable medium of claim 15 , wherein:

the frame-based data includes a frame and a corresponding frame time stamp;

the event-based data includes an output and a corresponding output time stamp; and

the output time stamp for the output corresponding to the event-based tracking point is synchronized with the frame time stamp for the frame corresponding to the frame-based tracking point.

19. The non-transitory computer readable medium of claim 15 , wherein the computer readable program code that when executed further causes at least one processor to:

determine a frame-based pose from the frame-based data;

determine an event-based pose from the event-based data; and

monitor the frame-based pose and the event-based pose to be within a tolerance.

20. The non-transitory computer readable medium of claim 19 , wherein:

the event-based camera captures a second output before the frame-based camera captures a second frame, and

the computer readable program code that when executed further causes at least one processor to:

identify a second event-based tracking point from the event-based data corresponding to the second output; and

synchronize the frame-based camera and the event-based camera based on the identified frame-based tracking point and the identified second event-based tracking point.

21. A device for tracking point synchronization comprising:

an inertial motion unit (IMU), configured to generate motion data based on an HMD movement;

at least one optical sensor; and

a processor coupled to the IMU and the at least one optical sensor, the processor configured to execute instructions to:

receive data from the IMU;

receive low resolution data and high resolution data at different framerates from the at least one optical sensor;

check whether data is available from a more precise source for each data point received;

update an offset for a less precise data source based on the more precise source when available; and

determine pose based on the updated offset.

22. A head mounted display (HMD) comprising:

an inertial motion unit (IMU), configured to generate motion data based on an HMD movement;

at least one outward facing optical sensor coupled to the HMD;

a processor coupled to the IMU and the at least one optical sensor, the processor configured to execute instructions to:

receive motion data from the IMU;

determine rate of pose change of the HMD based on the received motion data; and

adjust an optical tracking rate from the outward facing optical sensor based on the determined rate of pose change.

23. The HMD of claim 22 , further comprising:

a memory couple to the processor; and

wherein to adjust the optical tracking rate from the outward facing optical sensor based on the determined rate of pose change further comprises instructions executable by the processor to:

store in a circular buffer in the memory, input from the outward facing optical sensor; and;

perform pose calculations from the input of the outward facing optical sensor stored in the circular buffer at a calculation rate based on the determined rate of pose change.

24. The HMD of claim 22 , wherein:

the at least one outward facing optical sensor comprises an adjustable frame rate; and

to adjust the optical tracking rate from the outward facing optical sensor based on the determined rate of pose change further comprises instructions executable by the processor to:

adjust the adjustable frame rate of the at least one outward facing optical sensor based on the determined rate of pose change.

25. A head mounted display (HMD) comprising:

an outward facing frame-based camera coupled to the HMD;

an outward facing event-based camera coupled to the HMD in proximity to the frame-based camera;

an inertial motion unit (IMU), configured to generate motion data based on an HMD movement; and

a processor coupled to the frame-based camera, the event-based camera, and the IMU, the processor configured to execute instructions to:

receive motion data from the IMU;

select at least one of the frame-based camera or the event-based camera based on the motion data;

receive image data from the selected camera; and

process received image data to determine an HMD orientation.