Generating virtual notation surfaces with gestures in an augmented and/or virtual reality environment

1. A method, comprising:

displaying, on a display of a head mounted display (HMD) device, a virtual environment;

detecting, outside of the HMD, a first gesture input based on a detected first position of a first handheld electronic device, and a detected first position of a second handheld electronic device;

generating, in response to detection of the first gesture input, a virtual notation surface;

detecting, outside of the HMD, a second gesture input based on a detected second position of the first handheld electronic device and a detected second position of the second handheld electronic device;

displaying, in response to detection of the second gesture input, the virtual notation surface as a first virtual object in the virtual environment;

detecting a release of an actuation of a manipulation device of at least one of the first handheld electronic device or the second handheld electronic device;

in response to the detected release of the actuation of the manipulation device, setting at least one of a size or a shape of the virtual notation surface; and

displaying the virtual notation surface as a second virtual object in the virtual environment.

2. The method of claim 1 , wherein the first handheld electronic device is operably coupled with the HMD, and the second handheld electronic device is operably coupled with the HMD.

3. The method of claim 1 , wherein setting the at least one of the size or the shape of the virtual notation surface includes:

setting the at least one of the size or the shape of the virtual notation surface based on the detected second position of the first handheld electronic device relative to the detected first position of the first handheld electronic device, and on the detected second position of the second handheld electronic device relative to the detected first position of the second handheld electronic device.

4. The method of claim 3 , wherein setting the at least one of the size or the shape of the virtual notation surface also includes:

setting the at least one of the size or the shape of the virtual notation surface in response to detection of the release of the actuation of the manipulation device, the second position of the first handheld electronic device and the detection of the second position of the second handheld electronic device being detected at a point corresponding to the detection of the release of the actuation of the manipulation device.

5. The method of claim 1 , further comprising:

detecting, outside of the HMD, a third gesture input;

annotating, in response to detection of the third gesture input, the virtual notation surface, the virtual notation surface, including:

detecting and tracking movement of a marking implement; and

annotating the virtual notation surface based on the detected movement of the marking implement; and

displaying the annotated virtual notation surface as the second virtual object in the virtual environment.

6. The method of claim 5 , annotating the virtual notation surface based on the detected movement of the marking implement including:

detecting actuation of a manipulation device of the marking implement;

initiating tracking of the movement of the marking implement in response to the detected actuation of the manipulation device;

detecting release of the actuation of the manipulation device; and

terminating tracking of the movement of the marking implement in response to the detected release of the manipulation device.

7. The method of claim 6 , annotating the virtual notation surface based on the detected movement of the marking implement also including:

detecting a contour corresponding to the tracked movement of the marking implement; and

annotating the virtual notation surface with a contour corresponding to the detected contour of the movement of the marking implement.

8. A system, comprising:

a computing device configured to generate a virtual reality environment, the computing device including:

a memory storing executable instructions; and

a processor configured to execute the instructions to cause the computing device to:

display, on a display of a head mounted display (HMD) device, a virtual environment;

detect, outside of the HMD, a first gesture input based on a detected first position of a first handheld electronic device, and a detected first position of a second handheld electronic device;

generate, in response to detection of the first gesture input, a virtual notation surface;

detect, outside of the HMD, a second gesture input based on a detected second position of the first handheld electronic device and a detected second position of the second handheld electronic device;

display, in response to detection of the second gesture input, the virtual notation surface as a first virtual object in the virtual environment;

detect a release of an actuation of a manipulation device of at least one of the first handheld electronic device or the second handheld electronic device;

in response to the detected release of the actuation of the manipulation device, set at least one of a size or a shape of the virtual notation surface; and

display the virtual notation surface as a second virtual object in the virtual environment.

9. The system of claim 8 , wherein the first handheld electronic device is operably coupled with the HMD, and the second handheld electronic device is operably coupled with the HMD.

10. The system of claim 8 , wherein, in setting the at least one of the size or the shape of the virtual notation surface, the instructions cause the computing device to:

set the at least one of the size or the shape of the virtual notation surface based on the detected second position of the first handheld electronic device relative to the detected first position of the first handheld electronic device, and on the detected second position of the second handheld electronic device relative to the detected first position of the second handheld electronic device.

11. The system of claim 10 , wherein, in setting the at least one of the size or the shape of the virtual notation surface, the instructions cause the computing device to:

set the at least one of the size or the shape of the virtual notation surface in response to detection of the release of the actuation of the manipulation device, the second position of the first handheld electronic device and the detection of the second position of the second handheld electronic device being detected at a point corresponding to the detection of the release of the actuation of the manipulation device.

12. The system of claim 8 , wherein the instructions cause the computing device to:

detect, outside of the HMD, a third gesture input;

annotate, in response to detection of the third gesture input, the virtual notation surface, including:

detect and track movement of a marking implement; and

annotate the virtual notation surface based on the detected movement of the marking implement; and

display the annotated virtual notation surface as the second virtual object in the virtual environment.

13. The system of claim 12 , wherein, in annotating the virtual notation surface based on the detected movement of the marking implement, the instructions cause the computing device to:

detect actuation of a manipulation device of the marking implement;

initiate tracking of the movement of the marking implement in response to the detected actuation of the manipulation device;

detect release of the actuation of the manipulation device; and

terminate tracking of the movement of the marking implement in response to the detected release of the manipulation device.

14. The system of claim 13 , wherein, in annotating the virtual notation surface based on the detected movement of the marking implement, the instructions cause the computing device to:

detect a contour corresponding to the tracked movement of the marking implement; and

annotate the virtual notation surface with a contour corresponding to the detected contour of the movement of the marking implement.

15. A computer program product embodied on a non-transitory computer readable medium, the non-transitory computer readable medium having stored thereon a sequence of instructions which, when executed by a processor causes the processor to execute a method, the method comprising:

displaying, on a display of a head mounted display (HMD) device, a virtual environment;

detecting, outside of the HMD, a first gesture input based on a detected first position of a first handheld electronic device, and a detected first position of a second handheld electronic device;

generating, in response to detection of the first gesture input, a virtual notation surface;

detecting, outside of the HMD, a second gesture input based on a detected second position of the first handheld electronic device and a detected second position of the second handheld electronic device;

displaying, in response to detection of the second gesture input, the virtual notation surface as a first virtual object in the virtual environment;

detecting a release of an actuation of a manipulation device of at least one of the first handheld electronic device or the second handheld electronic device;

in response to the detected release of the actuation of the manipulation device, setting at least one of a size or a shape of the virtual notation surface; and

displaying the virtual notation surface as a second virtual object in the virtual environment.

16. The computer program product embodied on the non-transitory computer readable medium of claim 15 , detecting the first gesture input also including:

detecting a first actuation of a manipulation device of at least one of the first handheld electronic device or the second handheld electronic device; and

detecting the first gesture input based on the detected first position of the first handheld electronic device, the detected first position of the second handheld electronic device, and the detected first actuation of the manipulation device.

17. The computer program product embodied on the non-transitory computer readable medium of claim 16 , detecting the second gesture input including:

after detecting the first gesture input,

detecting a release of the first actuation of the manipulation device;

detecting the second position of the first handheld electronic device at a point at which the release of the first actuation of the manipulation device is detected;

detecting the second position of the second handheld electronic device at the point at which the release of the first actuation of the manipulation device is detected; and

setting the at least one of the size or the shape of the virtual notation surface based on

the detected second position of the first handheld electronic device relative to the detected first position of the first handheld electronic device, and

the detected second position of the second handheld electronic device relative to the detected first position of the second handheld electronic device.

18. The computer program product embodied on the non-transitory computer readable medium of claim 17 , further comprising:

detecting, outside of the HMD, a third gesture input; and

annotating, in response to detection of the third gesture input, the virtual notation surface including:

detecting a second actuation of the manipulation device;

tracking movement of at least one of the first handheld electronic device or the second handheld electronic device in response to the detected second actuation of the manipulation device;

detecting a release of the second actuation of the manipulation device;

terminating tracking of the movement of the at least one of the first handheld electronic device or the second handheld electronic device in response to the detected release of the second actuation of the manipulation device; and

annotating the virtual notation surface with a contour corresponding to the movement of the at least one of the first handheld electronic device or the second handheld electronic device tracked between detection of the second actuation of the manipulation device and detection of the release of the second actuation of the manipulation device.