Granted Patent B1

US 11,983,401 · App. 15/832,697 · Granted May 14, 2024

Inventors: Bingxin Ku (San Francisco, CA); Pohung Chen (Fremont, CA); Isaac L. Cohen (Oakland, CA); Paul A. Durdik (Foster City, CA)

Assignee: ULTRAHAPTICS IP TWO LIMITED, , United Kingd

G06F3/04883G06T19/006

Abstract

Claims (55)

1. A method including:

creating a real-time digital representation of a real-world physical environment in which a user of a head mounted device is positioned, the environment further comprising a 3D sensory space with a defined volume of interest;

generating, for continuous display by the head mounted device, a live video stream including the real-time digital representation of the real-world physical environment;

providing the user of the head mounted device with the generated live video stream;

identifying, by the head mounted device, a unique real-world marker that is present in the real-world physical environment from (i) the live video stream or (ii) the real-world physical environment;

identifying, by the head mounted device, a virtual interactive item, from a library of virtual interactive items, that corresponds to and provides information about the identified unique real-world marker;

generating, for display, 3D virtual imagery including (i) a virtual grid with a plurality of grid lines, (ii) virtual imagery corresponding to the identified virtual interactive item, and (iii) virtual imagery corresponding to a plurality of virtual items responsive to gestures in the 3D sensory space, wherein each generated virtual item is in visual correspondence with a different set of gridlines and wherein the generated 3D virtual imagery is superimposed, as a free-floating virtual modality in the real-world physical environment allowing the user to simultaneously view both the 3D virtual imagery and the real-world physical environment;

providing the user of the head mounted device with the generated 3D virtual imagery;

detecting, using images provided in the generated live video stream, a gesture in the defined volume of interest of the 3D sensory space by the user of the head mounted device;

interpreting the detected gesture as selecting one virtual item from the library of virtual interactive items;

generating, for display by the head mounted device, a menu associated with the selected virtual item;

providing the user of the head mounted device with the generated menu; and

modifying a presentation property of the selected virtual item in the virtual grid by augmenting the selected and displayed virtual item with additional graphics.

2. The method of claim 1 , further including superimposing, by the head mounted device, the identified virtual interactive item onto the identified unique real-world marker.

3. The method of claim 1 , wherein

the head mounted device includes one or more projectors that project imagery into the real-world physical environment and wherein the method further includes projecting, by the one or more projectors of the head mounted device, the identified virtual interactive item onto the identified unique real-world marker in the real-world physical environment, such that other individuals in the real-world physical environment can view the virtual interactive item that is projected into the real-world environment.

4. The method of claim 1 , wherein:

the identifying of the virtual interactive item identifies two or more virtual interactive items, from the library of virtual interactive items, that correspond to and provide information about the identified unique real-world marker; and

the identified two or more virtual interactive items are included in the 3D virtual imagery generated for display.

5. The method of claim 1 , wherein

the detected gesture is a scooping gesture in which a representation of a hand of a user appears to start from a position behind a virtual item and then proceed in a motion that appears to scoop up the virtual item from behind.

6. The method of claim 1 , wherein

the identified virtual interactive item is superimposed in place of the identified unique real-world marker, so as to replace the identified unique real-world marker from view of the user.

7. The method of claim 1 wherein

the unique real-world marker comprises at least one of a two- or three-dimensional barcode.

8. The method of claim 1 wherein

the unique real-world marker comprises an image of a real-world item.

9. A method including:

creating a real-time digital representation of a real-world physical environment in which a user of a head mounted device is positioned, the environment further comprising a 3D sensory space with a defined volume of interest;

generating, for continuous display by the head mounted device, a live video stream including the real-time digital representation of the real-world physical environment;

providing the user of the head mounted device with the generated live video stream;

identifying, by the head mounted device, a unique real-world marker that is present in the real-world physical environment from (i) the live video stream or (ii) the real-world physical environment;

identifying, by the head mounted device, a virtual interactive item, from a library of virtual interactive items, that corresponds to and provides information about the identified unique real-world marker;

generating, for display, 3D virtual imagery including (i) a virtual grid with a plurality of grid lines, (ii) virtual imagery corresponding to the identified virtual interactive item, and (iii) virtual imagery corresponding to a plurality of virtual items responsive to gestures in the 3D sensory space, wherein each generated virtual item is in visual correspondence with a different set of gridlines and wherein the generated 3D virtual imagery is superimposed, as a free-floating virtual modality in the real-world physical environment allowing the user to simultaneously view both the 3D virtual imagery and the real-world physical environment;

providing the user of the head mounted device with the generated 3D virtual imagery;

detecting, using images provided in the generated live video stream, a first sweep of a control object responsive to a first control gesture in the defined volume of interest of the 3D sensory space by the user of the head mounted device;

defining an extent of translation along a first axis of a virtual grid in proportion to length of the detected first sweep of the control object;

detecting, using images provided in the generated live video stream, a second sweep of the control object responsive to a second control gesture in the defined volume of interest of the 3D sensory space by the user of the head mounted device;

defining an extent of translation along a second axis of the virtual grid in proportion to length of the detected second sweep of the control object, wherein the second axis is perpendicular to the first axis;

automatically selecting a virtual item of the generated virtual items in the virtual grid at a terminal end of the detected second sweep; and

modifying a presentation property of the selected virtual item in the virtual grid by augmenting the selected and displayed virtual item with additional graphics.

10. A method including:

creating a real-time digital representation of a real-world physical environment in which a user of a head mounted device is positioned, the environment further comprising a 3D sensory space with a defined volume of interest;

generating, for continuous display by the head mounted device, a live video stream including the real-time digital representation of the real-world physical environment;

providing the user of the head mounted device with the generated live video stream;

identifying, by the head mounted device, a unique real-world marker that is present in the real-world physical environment from (i) the live video stream or (ii) the real-world physical environment;

identifying, by the head mounted device, a virtual interactive item, from a library of virtual interactive items, that corresponds to and provides information about the identified unique real-world marker;

generating, for display, 3D virtual imagery including (i) a virtual grid with a plurality of grid lines, (ii) virtual imagery corresponding to the identified virtual interactive item and (iii) virtual imagery corresponding to a plurality of virtual items responsive to gestures in the 3D sensory space, wherein each generated virtual item is in visual correspondence with a different set of gridlines and wherein the generated 3D virtual imagery is superimposed, as a free-floating virtual modality in the real-world physical environment allowing the user to simultaneously view both the 3D virtual imagery and the real-world physical environment;

providing the user of the head mounted device with the generated 3D virtual imagery;

detecting, using images provided in the generated live video stream, a horizontal sweep of a control object responsive to a first control gesture in the defined volume of interest of the 3D sensory space by the user of the head mounted device;

defining a horizontal extent of translation along a first axis of a virtual grid in proportion to length of the detected horizontal sweep of the control object;

detecting, using images provided in the generated live video stream, a vertical sweep of the control object responsive to a second control gesture in the defined volume of interest of the 3D sensory space by the user of the head mounted device;

defining a vertical extent of translation along a second axis of the virtual grid in proportion to length of the detected vertical sweep of the control object, wherein the second axis is perpendicular to the first axis;

automatically selecting virtual item of the generated virtual items in the virtual grid at a terminal end of detected the vertical sweep; and

modifying a presentation property of the selected virtual item in the virtual grid by augmenting the selected and displayed virtual item with additional graphics.

Assignments (14)

Continuity (2)

Cited By (4)

US 12,243,171 US 12,321,661 US 12,405,711 US 12,669,912