Training a neural network with representations of user interface devices
An example wearable display system is capable of determining a user interface (UI) event with respect to a virtual UI device (e.g., a button) and a pointer (e.g., a finger or a stylus) using a neural network. The wearable display system can render a representation of the UI device onto an image of the pointer captured when the virtual UI device is shown to the user and the user uses the pointer to interact with the virtual UI device. The representation of the UI device can include concentric shapes (or shapes with similar or the same centers of gravity) of high contrast. The neural network can be trained using training images with representations of virtual UI devices and pointers.
1. A wearable display system comprising:
an image capture device configured to capture an image comprising a pointer;
non-transitory computer-readable storage medium configured to store:
the image,
a virtual user interface (UI) device associated with the image at an image location on the image, and
a neural network for determining a UI event trained using:
an augmented training image associated with the virtual UI device, the augmented training image comprising a representation of the virtual UI device and a representation of a real-world pointer, and
a UI event with respect to the virtual UI device and the real-world pointer in the augmented training image;
a display configured to display the virtual UI device at a display location when the image is captured by the image capture device; and
a hardware processor in communication with the image capture device, the display, and the non-transitory computer-readable storage medium, the processor programmed by executable instructions to:
access the image captured by the image capture device;
render a representation of the virtual UI device onto the image at the image location;
determine, using the neural network, a UI event with respect to the pointer in the image and the virtual UI device associated with the image;
generate virtual content based on the UI event; and
cause the display to present the virtual content.
2. The wearable display system of claim 1 , wherein the representation of the virtual UI device comprises a plurality of shapes of high contrasts.
3. The wearable display system of claim 2 , wherein the plurality of shapes of high contrasts comprises a plurality of concentric shapes of high contrast.
4. The wearable display system of claim 2 , wherein the centers of gravity of shapes of the plurality of shapes are within a threshold distance of each other.
5. The wearable display system of claim 4 , wherein the threshold distance is 0.
6. The wearable display system of claim 2 ,
wherein a first shape of the plurality of shapes is associated with a first color,
wherein a second shape of the plurality of shapes is associated with a second color, and
wherein the first shape is adjacent to the second shape.
7. The wearable display system of claim 6 , wherein the first color is black, and wherein the second color is white.
8. The wearable display system of claim 6 , wherein the second shape is partially transparent.
9. The wearable display system of claim 8 ,
wherein a first region of the second shape is associated with a first transparency value, and
wherein a second region of the second shape is associated with a second transparency value.
10. The wearable display system of claim 9 ,
wherein the second region of the second shape is further away from the first region of the second shape, and
wherein the second transparency value is smaller than the first transparency value.
11. The wearable display system of claim 6 , wherein the second shape is within the first shape.
12. The wearable display system of claim 6 ,
wherein a third shape of the plurality of shapes is associated with a third color, and
wherein the first shape is adjacent to the first shape.
13. The wearable display system of claim 12 , wherein third color is white.
14. The wearable display system of claim 12 , wherein the first shape is within the third shape.
15. The wearable display system of claim 1 , wherein the UI event corresponds to a state of a plurality of states of the virtual UI device.
16. The wearable display system of claim 15 , wherein the plurality of states comprises activation or non-activation of the virtual UI device.
17. The wearable display system of claim 1 , wherein the virtual UI device is selected from a group comprising of: a button, an updown, a spinner, a picker, a radio button, a radio button list, a checkbox, a picture box, a checkbox list, a dropdown list, a dropdown menu, a selection list, a list box, a combo box, a textbox, a slider, a link, a keyboard key, a switch, a slider, a touch surface, or a combination thereof.
18. The wearable display system of claim 1 , wherein the pointer comprises an object associated with a user or a part of the user.
19. The wearable display system of claim 18 ,
wherein the object associated with the user comprises a stylus, a pen, a pencil, a marker, a highlighter, or a combination thereof, and
wherein the part of the user comprises a finger of the user.
20. A computerized method, performed by a computing system having one or more hardware computer processors and one or more non-transitory computer readable storage device storing software instructions executable by the computing system to perform the computerized method comprising:
accessing an image captured by an image capture device of an augmented reality headset, the image including a pointer;
rendering, on a display of the augmented reality headset, an augmented image comprising the image and a representation of a virtual UI device;
applying, to the augmented image, a neural network trained using:
an augmented training image associated with the virtual UI device, the augmented training image comprising a representation of the virtual UI device and a representation of a real-world pointer, and
a UI event with respect to the virtual UI device and the real-world pointer in the augmented training image;
detecting, based on application of the neural network, the UI event with respect to the pointer in the image and the virtual UI device associated with the image;
generating virtual content based on the UI event; and
causing the display to present the virtual content.