Granted Patent B2

US 12,357,911 · App. 18/407,308 · Granted Jul 15, 2025

Inventors: Ilteris Canberk (Marina Del Rey, CA); Dmytro Kucher (Odessa, UA); Dmytro Hovorov (Luhansk, UA); Artur Sydoran (Uzhhorod, UA)

Assignee: Snap Inc.

A63F13/428A63F13/213A63F13/26A63F13/537G06F3/017G06F3/0425G06F3/04817G06T11/60A63F2300/8082G06T2210/12G06T2210/21

Abstract

Claims (132)

1. A method of presenting an interactive game in augmented reality, the method comprising:

capturing frames of video data with a camera coupled to an eyewear device comprising a processor, a memory, a hand tracking utility, and a display;

presenting on the display a virtual field of play comprising a distal location and a scoring location, wherein the virtual field of play is persistently viewable as an overlay relative to a physical environment, independent of movement of the eyewear device relative to the physical environment,

presenting on the display a first icon in apparent motion from the distal location toward the scoring location on the virtual field of play;

detecting in the frames of video data, with the hand tracking utility, a final hand shape;

selecting a final virtual game piece from a set of virtual game pieces based on the final hand shape;

presenting on the display the final virtual game piece as an overlay relative to the final hand shape;

detecting a final hand location associated with the final hand shape relative to the scoring location; and

incrementing a game score on the display if the final virtual game piece matches the first icon and the final hand location matches the scoring location.

2. The method of claim 1 , wherein detecting the final hand location further comprises:

presenting on the display a game clock configured to increment from a start time to a final time;

detecting the final hand location associated with the final time; and

incrementing the game score based on the final time, the final virtual game piece, and the final hand location.

3. The method of claim 1 , wherein incrementing the game score further comprises:

detecting an opposing final virtual game piece, and

incrementing the game score based on a comparison of the final virtual game piece to the opposing final virtual game piece.

4. The method of claim 1 , further comprising:

presenting on the display a game clock configured to increment from a start time to a final time;

detecting in the captured frames of video data, with the hand tracking utility and before the final time, a preliminary hand shape;

presenting on the display a bounding box near the preliminary hand shape; and

presenting on the display a preliminary virtual game piece based on the detected preliminary hand shape.

5. The method of claim 1 , further comprising:

presenting on the display a game clock configured to increment from a start time to a final time;

synchronizing the game clock with an opponent device comprising an opponent display, such that the game clock presents and increments simultaneously on the display and the opponent display;

initiating the game clock at the start time on both the displays in response to detecting a start signal in the captured frames of video data;

detecting an opposing final virtual game piece; and

presenting on the display the opposing final virtual game piece within the virtual field of play and spaced apart from the final virtual game piece.

6. The method of claim 1 , further comprising:

presenting on the display a game clock configured to increment from a start time to a final time;

presenting on the display a first key image at the scoring location;

presenting on the display the virtual field of play, wherein the virtual field of play comprises a first lane associated with a virtual scroll extending from the distal location toward the scoring location;

presenting on the display the first icon in apparent motion along the near first lane from the distal location toward the first key image at the scoring location, wherein the first icon persistently matches the first key image;

detecting a final hand location relative to the first lane, wherein the final time is further associated with an apparent collision between the first icon and the first key image;

determining whether the detected final hand location is a location match relative to the apparent collision in the first lane;

determining whether the detected final hand shape is a shape match relative to the first key image; and

in response to detecting both the shape match and the location match, incrementing the game score by one.

7. The method of claim 1 , wherein selecting the final virtual game piece further comprises:

presenting on the display a bounding box near the final hand location, wherein the bounding box comprises a subset of depth information associated with a plurality of pixels;

identifying a set of hand landmarks associated with the final hand shape based on the subset of depth information; and

selecting a candidate hand gesture from a library of hand gestures based on the set of hand landmarks.

8. The method of claim 7 , wherein the library of hand gestures and the set of virtual game pieces comprises:

a first gesture associated with a virtual fist-like game piece, a fist-shaped key image, and a fist-shaped icon;

an open-hand gesture associated with a virtual open-hand game piece, a hand-shaped key image, and a hand-shaped icon;

a two-finger vee gesture associated with a virtual two-finger vee game piece, a vee-shaped key image, and a vee-shaped icon;

a pointing gesture associated with a virtual pointing-finger game piece, a pointing key image, and a pointing key icon; and

a two-finger horns gesture associated with a virtual two-finger horns game piece, a two-finger key image, and a two-finger key icon.

9. An interactive game presentation system, comprising:

an eyewear device comprising a processor, a memory, a camera, a hand tracking utility, and a display;

programming in the memory, wherein execution of the programming by the processor configured the eyewear device to perform functions, including functions to:

capture frames of video data with a camera coupled to an eyewear device comprising a processor, a memory, a hand tracking utility, and a display;

presenting on the display a virtual field of play comprising a distal location and a scoring location, wherein the virtual field of play is persistently viewable as an overlay relative to a physical environment, independent of movement of the eyewear device relative to the physical environment;

present on the display a first icon in apparent motion from the distal location toward the scoring location on the virtual field of play;

detect in the frames of video data, with the hand tracking utility, a final hand shape;

select a final virtual game piece from a set of virtual game pieces based on the final hand shape;

present on the display the final virtual game piece as an overlay relative to the final hand shape;

detect a final hand location associated with the final hand shape relative to the scoring location; and

increment a game score on the display if the final virtual game piece matches the first icon and the final hand location matches the scoring location.

10. The system of claim 9 , wherein the execution of the programming further configures the eyewear device to:

present on the display a game clock configured to increment from a start time to a final time;

detect the final hand location associated with the final time; and

increment the game score based on the final time, the final virtual game piece, and the final hand location.

11. The system of claim 9 , wherein the execution of the programming further configures the eyewear device to:

present on the display a game clock configured to increment from a start time to a final time;

detect in the captured frames of video data, with the hand tracking utility and before the final time, a preliminary hand shape;

present on the display a bounding box near the preliminary hand shape; and

present on the display a preliminary virtual game piece based on the detected preliminary hand shape.

12. The system of claim 9 , wherein the execution of the programming further configures the eyewear device to:

present on the display a game clock configured to increment from a start time to a final time;

synchronize the game clock with an opponent device comprising an opponent display, such that the game clock presents and increments simultaneously on the display and the opponent display;

initiate the game clock at the start time on both the displays in response to detecting a start signal in the captured frames of video data;

detect an opposing final virtual game piece; and

present on the display the opposing final virtual game piece within the virtual field of play and spaced apart from the final virtual game piece.

13. The system of claim 9 , wherein the execution of the programming further configures the eyewear device to:

present on the display a game clock configured to increment from a start time to a final time;

present on the display a first key image at the scoring location;

present on the display the virtual field of play, wherein the virtual field of play comprises a first lane associated with a virtual scroll extending from the distal location toward the scoring location;

present on the display the first icon in apparent motion along the first lane from the distal location toward the first key image at the scoring location, wherein the first icon persistently matches the first key image;

detect a final hand location relative to the first lane, wherein the final time is further associated with an apparent collision between the first icon and the first key image;

determine whether the detected final hand location is a location match relative to the apparent collision in the first lane;

determine whether the detected final hand shape is a shape match relative to the first key image; and

in response to detecting both the shape match and the location match, increment the game score by one.

14. The system of claim 9 , wherein the execution of the programming further configures the eyewear device to:

present on the display a bounding box near the final hand location, wherein the bounding box comprises a subset of depth information associated with a plurality of pixels;

identify a set of hand landmarks associated with the final hand shape based on the subset of depth information; and

select a candidate hand gesture from a library of hand gestures based on the set of hand landmarks, wherein the library of hand gestures and the set of virtual game pieces comprises:

a first gesture associated with a virtual fist-like game piece, a fist-shaped key image, and a fist-shaped icon;

an open-hand gesture associated with a virtual open-hand game piece, a hand-shaped key image, and a hand-shaped icon;

a two-finger vee gesture associated with a virtual two-finger vee game piece, a vee-shaped key image, and a vee-shaped icon;

a pointing gesture associated with a virtual pointing-finger game piece, a pointing key image, and a pointing key icon; and

a two-finger horns gesture associated with a virtual two-finger horns game piece, a two-finger key image, and a two-finger key icon.

15. A non-transitory computer-readable medium storing program code which, when executed, is operative to cause an electronic processor to perform the steps of:

capturing frames of video data with a camera coupled to an eyewear device comprising a processor, a memory, a hand tracking utility, and a display;

presenting on the display a virtual field of play comprising a distal location and a scoring location, wherein the virtual field of play is persistently viewable as an overlay relative to a physical environment, independent of movement of the eyewear device relative to the physical environment;

presenting on the display a first icon in apparent motion from the distal location toward the scoring location on the virtual field of play;

detecting in the frames of video data, with the hand tracking utility, a final hand shape;

selecting a final virtual game piece from a set of virtual game pieces based on the final hand shape;

presenting on the display the final virtual game piece as an overlay relative to the final hand shape;

detecting a final hand location associated with the final hand shape relative to the scoring location; and

incrementing a game score on the display if the final virtual game piece matches the first icon and the final hand location matches the scoring location.

16. The non-transitory computer-readable medium storing program code of claim 15 , wherein the program code, when executed, is operative to cause an electronic processor to perform the further steps of:

presenting on the display a game clock configured to increment from a start time to a final time;

detecting the final hand location associated with the final time; and

incrementing the game score based on the final time, the final virtual game piece, and the final hand location.

17. The non-transitory computer-readable medium storing program code of claim 15 , wherein the program code, when executed, is operative to cause an electronic processor to perform the further steps of:

presenting on the display a game clock configured to increment from a start time to a final time;

detecting in the captured frames of video data, with the hand tracking utility and before the final time, a preliminary hand shape;

presenting on the display a bounding box near the preliminary hand shape; and

presenting on the display a preliminary virtual game piece based on the detected preliminary hand shape.

18. The non-transitory computer-readable medium storing program code of claim 15 , wherein the program code, when executed, is operative to cause an electronic processor to perform the further steps of:

presenting on the display a game clock configured to increment from a start time to a final time;

synchronizing the game clock with an opponent device comprising an opponent display, such that the game clock presents and increments simultaneously on the display and the opponent display;

initiating the game clock at the start time on both the displays in response to detecting a start signal in the captured frames of video data;

detecting an opposing final virtual game piece; and

presenting on the display the opposing final virtual game piece within the virtual field of play and spaced apart from the final virtual game piece.

19. The non-transitory computer-readable medium storing program code of claim 15 , wherein the program code, when executed, is operative to cause an electronic processor to perform the further steps of:

presenting on the display a game clock configured to increment from a start time to a final time;

presenting on the display a first key image at the scoring location;

presenting on the display the virtual field of play, wherein the virtual field of play comprises a first lane associated with a virtual scroll extending from the distal location toward the scoring location;

presenting on the display the first icon in apparent motion along the first lane from the distal location toward the first key image at the scoring location, wherein the first icon persistently matches the first key image;

detecting a final hand location relative to the first lane, wherein the final time is further associated with an apparent collision between the moving first icon and the first key image;

determining whether the detected final hand location is a location match relative to the apparent collision in the first lane;

determining whether the detected final hand shape is a shape match relative to the first key image; and

in response to detecting both the shape match and the location match, incrementing the game score by one.

20. The non-transitory computer-readable medium storing program code of claim 15 , wherein the program code, when executed, is operative to cause an electronic processor to perform the further steps of:

presenting on the display a bounding box near the final hand location, wherein the bounding box comprises a subset of depth information associated with a plurality of pixels;

identifying a set of hand landmarks associated with the final hand shape based on the subset of depth information; and

selecting a candidate hand gesture from a library of hand gestures based on the set of hand landmarks, wherein the library of hand gestures and the set of virtual game pieces comprises:

a first gesture associated with a virtual fist-like game piece, a fist-shaped key image, and a fist-shaped icon;

an open-hand gesture associated with a virtual open-hand game piece, a hand-shaped key image, and a hand-shaped icon;

a two-finger vee gesture associated with a virtual two-finger vee game piece, a vee-shaped key image, and a vee-shaped icon;

a pointing gesture associated with a virtual pointing-finger game piece, a pointing key image, and a pointing key icon; and

a two-finger horns gesture associated with a virtual two-finger horns game piece, a two-finger key image, and a two-finger key icon.

Assignments (1)

Continuity (2)

References Cited (206)