Hand presence over keyboard inclusiveness
In one embodiment, a method includes a computer system accessing an image of a physical environment of a user, the image being associated with a perspective of the user and depicting a physical input device and a physical hand of the user, determining a pose of the physical input device, generating a three-dimensional model representing the physical hand of the user, generating an image mask by projecting the three-dimensional model onto an image plane associated with the perspective of the user, generating, by applying the image mask to the image, a cropped image depicting at least the physical hand of the user in the image, rendering, based on the perspective of the user and the pose of the physical input device, a virtual input device to represent the physical input device, and displaying the cropped image depicting at least the physical hand over the rendered virtual input device.
1. A method comprising, by a computing system:
accessing an image of a physical environment of a user, the image being associated with a perspective of the user and depicting a physical input device and a physical hand of the user;
determining a pose of the physical input device;
generating a three-dimensional model representing the physical hand of the user;
generating an image mask by projecting the three-dimensional model onto an image plane associated with the perspective of the user, wherein the image mask comprises a buffer region surrounding a contour of a projection of the three-dimensional model on the image plane;
generating, by applying the image mask to the image, a cropped image depicting at least the physical hand of the user in the image;
rendering, based on the perspective of the user and the pose of the physical input device, a virtual input device to represent the physical input device; and
displaying the cropped image depicting at least the physical hand of the user over the rendered virtual input device.
2. The method of claim 1 , wherein the image is generated by projecting image data captured from one or more second perspectives of one or more cameras of a head-mounted device onto the image plane associated with the perspective of the user, the one or more second perspectives of the one or more cameras being different from the perspective of the user.
3. The method of claim 2 , wherein the image data captured by the one or more cameras is used to determine the pose of the physical input device and generate the three-dimensional model representing the physical hand of the user.
4. The method of claim 1 , wherein at least the buffer region comprises alpha-blending values.
5. The method of claim 1 , further comprising:
determining that a contrast between the physical input device and the physical hand depicted in the image is lower than a predetermined threshold; and
modifying the image to increase the contrast between the physical input device and the physical hand depicted in the image.
6. The method of claim 1 , further comprising:
determining that a contrast between the physical input device depicted in the image and the virtual input device representing the physical input device is greater than a predetermined threshold; and
modifying the image to decrease the contrast between the physical input device depicted in the image and the virtual input device representing the physical input device.
7. The method of claim 1 , further comprising:
determining that one or more points in the image depicting the physical hand are associated with a shadow of the physical hand; and
modifying the image to exclude the one or more points associated with the shadow of the physical hand.
8. The method of claim 1 , wherein the cropped image is displayed in response to a determination that the three-dimensional model representing the physical hand of the user is within a predefined bounding volume.
9. The method of claim 1 , further comprising:
rendering virtual labels for keys of the virtual input device; and
displaying the virtual labels over both the cropped image depicting the physical hand of the user and the rendered virtual input device.
10. A system comprising: one or more processors; and a memory coupled to the processors comprising instructions executable by the processors, the processors operable when executing the instructions to:
access an image of a physical environment of a user, the image being associated with a perspective of the user and depicting a physical input device and a physical hand of the user;
determine a pose of the physical input device;
generate a three-dimensional model representing the physical hand of the user;
generate an image mask by projecting the three-dimensional model onto an image plane associated with the perspective of the user, wherein the image mask comprises a buffer region surrounding a contour of a projection of the three-dimensional model on the image plane;
generate, by applying the image mask to the image, a cropped image depicting at least the physical hand of the user in the image;
render, based on the perspective of the user and the pose of the physical input device, a virtual input device to represent the physical input device; and
display the cropped image depicting at least the physical hand of the user over the rendered virtual input device.
11. The system of claim 10 , wherein the image is generated by projecting image data captured by one or more cameras of a head-mounted device onto the image plane associated with the perspective of the user.
12. The system of claim 11 , wherein the image data captured by the one or more cameras is used to determine the pose of the physical input device and generate the three-dimensional model representing the physical hand of the user.
13. The system of claim 10 , wherein the image mask comprises a buffer region surrounding a contour of a projection of the three-dimensional model on the image plane, and wherein at least the buffer region comprises alpha-blending values.
14. The system of claim 10 , wherein the processors are further operable when executing the instructions to:
determine that a contrast between the physical input device and the physical hand depicted in the image is lower than a predetermined threshold; and
modify the image to increase the contrast between the physical input device and the physical hand depicted in the image.
15. One or more computer-readable non-transitory storage media embodying software that is operable when executed to:
access an image of a physical environment of a user, the image being associated with a perspective of the user and depicting a physical input device and a physical hand of the user;
determine a pose of the physical input device;
generate a three-dimensional model representing the physical hand of the user;
generate an image mask by projecting the three-dimensional model onto an image plane associated with the perspective of the user, wherein the image mask comprises a buffer region surrounding a contour of a projection of the three-dimensional model on the image plane;
generate, by applying the image mask to the image, a cropped image depicting at least the physical hand of the user in the image;
render, based on the perspective of the user and the pose of the physical input device, a virtual input device to represent the physical input device; and
display the cropped image depicting at least the physical hand of the user over the rendered virtual input device.
16. The media of claim 15 , wherein the image is generated by projecting image data captured by one or more cameras of a head-mounted device onto the image plane associated with the perspective of the user.
17. The media of claim 16 , wherein the image data captured by the one or more cameras is used to determine the pose of the physical input device and generate the three-dimensional model representing the physical hand of the user.
18. The media of claim 15 , wherein the image mask comprises a buffer region surrounding a contour of a projection of the three-dimensional model on the image plane, and wherein at least the buffer region comprises alpha-blending values.
19. The media of claim 15 , wherein the software is further operable when executed to:
determine that a contrast between the physical input device and the physical hand depicted in the image is lower than a predetermined threshold; and
modify the image to increase the contrast between the physical input device and the physical hand depicted in the image.