Calibration of mobile electronic devices connected to headsets wearable by users
A mobile electronic device is provided for use with a headset. A camera outputs digital pictures of a portion of the headset. A display device displays information for viewing by a user wearing the headset. A processor retrieves calibration parameters that characterize at least a pose of the camera relative to the display device, and processes a digital picture from the camera to identify a pose of an optically identifiable feature within the digital picture. A pose of the mobile electronic device is identified relative to the holder based on the identified pose of the optically identifiable feature within the digital picture and based on at least the pose of the camera relative to the display device as characterized by the calibration parameters. The processor controls where graphical objects are rendered on the display device based on the identified pose of the mobile electronic device relative to the holder.
1. An apparatus for use with a headset, the apparatus comprising:
a camera configured to output digital pictures and arranged to view a portion of the headset;
a display device arranged to display information for viewing by a user wearing the headset;
a processor operationally connected to the camera and to the display device; and
a memory storing program code that is executed by the processor to perform operations comprising:
processing a digital picture from the camera to identify a pose relative to the camera of an optically identifiable feature of the headset within the digital picture; and
controlling where graphical objects are rendered on the display device based on the identified pose relative to the camera of the optically identifiable feature within the digital picture.
2. The apparatus of claim 1 , wherein the operations further comprise controlling where graphical objects are rendered on the display device further based on a defined relationship between a pose of the camera and a pose of the display device.
3. The apparatus of claim 1 , wherein the operation of controlling where graphical objects are rendered on the display device based on the identified pose relative to the camera of the optically identifiable feature within the digital picture comprises:
identifying a pose of the apparatus relative to a holder, configured to releasably retain the apparatus in a defined pose relative to the headset, based on the identified pose relative to the camera of the optically identifiable feature within the digital picture and based on a defined relationship between a pose of the camera and a pose of the apparatus; and
controlling where graphical objects are rendered on the display device based on the identified pose of the apparatus relative to the holder.
4. The apparatus of claim 1 , wherein the optically identifiable feature of the headset is configured to have a moveable pose relative to the display device.
5. The apparatus of claim 1 , wherein the operations further comprise:
determining a pose of the optically identifiable feature of the headset within a previous digital picture that was captured by the camera during a previous calibration process,
wherein the operation to process the digital picture from the camera to identify the pose of the optically identifiable feature of the headset within the digital picture is performed during a present calibration process, and
further comprising identifying pose of the optically identifiable feature of the headset relative to the display device, based on comparison of the identified pose of the optically identifiable feature of the headset within the digital picture that was captured during the present calibration process to the pose of the optically identifiable feature of the headset within the previous digital picture that was captured during the previous calibration process.
6. The apparatus of claim 1 , wherein the operations further comprise:
displaying a calibration pattern on the display device for the camera to capture a portion of the calibration pattern within the digital picture by reflection of the portion of the calibration pattern from a surface of the headset and/or from an eye of the user, and
the operation to process the digital picture from the camera to identify the pose relative of the camera of the optically identifiable feature within the digital picture, comprises processing the digital picture to identify the pose of a defined feature within the portion of the calibration pattern.
7. The apparatus of claim 1 , wherein the operations further comprise:
obtaining distortion parameters of the camera affecting pose of the defined feature within the digital picture,
wherein the operation to process the digital picture from the camera to identify the pose relative to the camera of the optically identifiable feature within the digital picture compensates for the distortion parameters of the camera.
8. The apparatus of claim 1 , wherein the operations further comprise:
obtaining headset-to-eye parameters identifying in at least a two-dimensional space distances between at least one location on the headset and at least one eye of the user,
wherein the operation to control where graphical objects are rendered on the display device is further performed based on the headset-to-eye parameters.
9. The apparatus of claim 1 , wherein the operations further comprise:
obtaining off-axis projection parameters identifying in at least a two-dimensional space distances between at least one location on the display device and at least one eye of the user,
wherein the operation to control where graphical objects are rendered on the display device is further performed based on the off-axis projection parameters.
10. The apparatus of claim 1 , further comprising:
generating a camera-to-headset transform that relates a pose of the camera to an optically identifiable feature of the headset within the digital picture;
obtaining a camera-to-display transform that relates the pose of the camera to a pose of the display device; and
generating a display-to-headset transform based on the camera-to-headset transform and the camera-to-display transform; and
the operation to control where graphical objects are rendered on the display device based on the identified pose relative to the camera of the optically identifiable feature within the digital picture, is performed based on the display-to-headset transform.
11. The apparatus of claim 10 , wherein the operations further comprise:
obtaining distortion parameters of the camera affecting pose of the optically identifiable feature of the headset within the digital picture,
wherein the operation to generate the camera-to-headset transform compensates for the distortion parameters of the camera.
12. The apparatus of claim 10 , wherein the operation to obtain the camera-to-display transform that relates the pose of the camera to the pose of the display device, comprises:
processing the digital picture or another digital picture from the camera to identify from a reflected image, which is generated by an image from the display device being reflected by a reflective surface of the headset to the camera, distances between the camera and the display device relative to at least a two-dimensional coordinate system; and
generating the camera-to-display transform based on the distances between the camera and the display device.
13. The apparatus of claim 12 , wherein the operation to generate the camera-to-display transform based on the distances between the camera and the display device, comprises:
obtaining data characterizing distances between a holder, configured to releasably retain the apparatus in a defined pose relative to the headset, and the reflective surface of the headset that produced the reflected image; and
generating the camera-to-display transform based on scaling the distances between the camera and the display device identified in the reflected image based on the data characterizing the distances between the holder and the reflective surface of the headset.
14. The apparatus of claim 1 , wherein the operations further comprise:
responsive to the identified pose relative to the camera of the optically identifiable feature within the digital picture exceeding an offset threshold value,
determining a direction and a distance that the apparatus is to be moved relative to a holder, configured to releasably retain the apparatus in a defined pose relative to the headset, so the display device is within a threshold distance of a defined location on the holder, and
displaying a message on the display device that indicates the direction and the distance that the apparatus is to be moved.
15. The apparatus of claim 1 , wherein the operations further comprise:
determining a pose of the apparatus relative to a holder configured to releasably retain the apparatus in a defined pose relative to the headset,
responsive to the pose of the apparatus relative to the holder exceeding an offset threshold value,
determining a direction that the apparatus is be moved relative to the holder so the display device is within a threshold distance of a defined location on the holder,
displaying a message on the display device that indicates the direction that the apparatus should be moved, and
repeating the determination of the direction and the display of another message that indicates the direction that the apparatus should be moved until the apparatus is within the threshold distance of the defined location on the holder.
16. The apparatus of claim 1 , wherein the operations further comprise:
displaying a calibration pattern on the display device for the camera to capture a portion of the calibration pattern within the digital picture or another digital picture by reflection of the portion of the calibration pattern from a surface of the headset and/or from an eye of the user; and
determining size of the display device and/or pixel density of the display device based on a determination of size of a feature within the portion of the calibration pattern,
wherein the controlling of where graphical objects are rendered on the display device is further based on the size of the display device and/or the pixel density of the display device.
17. The apparatus of claim 1 , wherein the operations further comprise:
operating the camera to capture a reflection of an image that is displayed by the display device and reflected by a surface of the headset and/or by an eye of the user; and
determining shape of the display device based on a determination of shape of the reflected image;
wherein the controlling of where graphical objects are rendered on the display device is further based on the shape of the display device.
18. A method of performing operations by a processor of an apparatus that includes a camera and a display device for use with a headset, the method comprising:
processing a digital picture from the camera to identify a pose relative to the camera of an optically identifiable feature of the headset within the digital picture; and
controlling where graphical objects are rendered on the display device based on the identified pose relative to the camera of the optically identifiable feature within the digital picture.
19. A computer program product for a processor of an apparatus that includes a camera and a display device for use with a headset, the computer program product comprising:
a non-transitory computer readable medium storing program code executable by the processor of the apparatus to perform operations comprising:
processing a digital picture from the camera to identify a pose relative to the camera of an optically identifiable feature of the headset within the digital picture; and
controlling where graphical objects are rendered on the display device based on the identified pose relative to the camera of the optically identifiable feature within the digital picture.
20. The computer program product of claim 19 , wherein the operations further comprise controlling where graphical objects are rendered on the display device further based on a defined relationship between a pose of the camera and a pose of the display device.
21. The computer program product of claim 19 , wherein the operation of controlling where graphical objects are rendered on the display device based on the identified pose relative to the camera of the optically identifiable feature within the digital picture comprises:
identifying a pose of the apparatus relative to a holder, configured to releasably retain the apparatus in a defined pose relative to the headset, based on the identified pose relative to the camera of the optically identifiable feature within the digital picture and based on a defined relationship between a pose of the camera and a pose of the apparatus; and
controlling where graphical objects are rendered on the display device based on the identified pose of the apparatus relative to the holder.
22. The computer program product of claim 19 , wherein the optically identifiable feature of the headset is configured to have a moveable pose relative to the display device.
23. The computer program product of claim 19 , wherein the operations further comprise:
determining a pose of the optically identifiable feature of the headset within a previous digital picture that was captured by the camera during a previous calibration process,
wherein the operation to process the digital picture from the camera to identify the pose of the optically identifiable feature of the headset within the digital picture is performed during a present calibration process, and
further comprising identifying pose of the apparatus relative to a holder, configured to releasably retain the apparatus in a defined pose relative to the headset, based on comparison of the identified pose of the optically identifiable feature of the headset within the digital picture that was captured during the present calibration process to the pose of the optically identifiable feature of the headset within the previous digital picture that was captured during the previous calibration process.
24. The computer program product of claim 19 , wherein the operations further comprise:
displaying a calibration pattern on the display device for the camera to capture a portion of the calibration pattern within the digital picture by reflection of the portion of the calibration pattern from a surface of the headset and/or from an eye of the user, and
the operation to process the digital picture from the camera to identify the pose relative of the camera of the optically identifiable feature within the digital picture, comprises processing the digital picture to identify the pose of a defined feature within the portion of the calibration pattern.
25. The computer program product of claim 19 , wherein the operations further comprise:
obtaining distortion parameters of the camera affecting pose of the defined feature within the digital picture,
wherein the operation to process the digital picture from the camera to identify the pose relative to the camera of the optically identifiable feature within the digital picture compensates for the distortion parameters of the camera.
26. The computer program product of claim 19 , wherein the operations further comprise:
obtaining headset-to-eye parameters identifying in at least a two-dimensional space distances between at least one location on the headset and at least one eye of the user,
wherein the operation to control where graphical objects are rendered on the display device is further performed based on the headset-to-eye parameters.
27. The computer program product of claim 19 , wherein the operations further comprise:
obtaining off-axis projection parameters identifying in at least a two-dimensional space distances between at least one location on the display device and at least one eye of the user,
wherein the operation to control where graphical objects are rendered on the display device is further performed based on the off-axis projection parameters.
28. The computer program product of claim 19 , further comprising:
generating a camera-to-headset transform that relates a pose of the camera to an optically identifiable feature of the headset within the digital picture;
obtaining a camera-to-display transform that relates the pose of the camera to a pose of the display device; and
generating a display-to-headset transform based on the camera-to-headset transform and the camera-to-display transform; and
the operation to control where graphical objects are rendered on the display device based on the identified pose relative to the camera of the optically identifiable feature within the digital picture, is performed based on the display-to-headset transform.
29. The computer program product of claim 28 , wherein the operation to obtain the camera-to-display transform that relates the pose of the camera to the pose of the display device, comprises:
processing the digital picture or another digital picture from the camera to identify from a reflected image, which is generated by an image from the display device being reflected by a reflective surface of the headset to the camera, distances between the camera and the display device relative to at least a two-dimensional coordinate system; and
generating the camera-to-display transform based on the distances between the camera and the display device.
30. The computer program product of claim 19 , wherein the operations further comprise:
operating the camera to capture a reflection of an image that is displayed by the display device and reflected by a surface of the headset and/or by an eye of the user; and
determining shape of the display device based on a determination of shape of the reflected image;
wherein the controlling of where graphical objects are rendered on the display device is further based on the shape of the display device.