Granted Patent B2

US 12,051,163 · App. 17/895,449 · Granted Jul 30, 2024

Inventors: Ilteris Kaan Canberk (Marina Del Rey, CA); Matthew Hallberg (Los Angeles, CA); Matthew Saunders (Toronto, CA); Daria Skrypnyk (Kyiv, UA); Lien Le Hong Tran (Santa Monica, CA)

Assignee: Snap Inc.

G06T19/006G06F3/011G06T13/40

Abstract

Claims (64)

1. A method comprising:

establishing, by one or more processors of an augmented reality (AR) device, a communication session with an external client device;

receiving a communication from the external client device indicating that a whole body of a user has been detected within view of a camera of the external client device;

in response to receiving the communication, presenting an AR notification to inform the user that the AR device is ready to control a first AR object;

causing, by the AR device, overlay of the first AR object on a real-world environment being viewed using the AR device;

receiving, by the AR device, interaction data from the external client device representing movement of the user determined by the external client device; and

in response to receiving the interaction data from the external client device, modifying the first AR object by the AR device.

2. The method of claim 1 , wherein the external client device comprises a smartphone, and wherein the AR device comprises an AR eyewear device.

3. The method of claim 1 , wherein the external client device comprises an external vision system, the method further comprising:

presenting by the external client device one or more images depicting the user to enable the user to position the user in a way that allows the camera of the external client device to capture images of an entire body of the user; and

presenting by the external client device an indication on a screen in response to detecting by the external client device the whole body of the user, the external client device transmitting the communication in response to detecting by the external client device the whole body of the user.

4. The method of claim 1 , wherein modifying the first AR object comprises animating an avatar.

5. The method of claim 4 , wherein animating the avatar comprises:

determining that the interaction data comprises body movement information; and

generating a whole body avatar as the avatar in response to determining that the movement information includes body movement information.

6. The method of claim 5 , further comprising:

updating one or more body parts of the avatar based on the body movement information received from the external client device.

7. The method of claim 1 , wherein the external client device comprises an external vision system of a messaging application implemented by a mobile device coupled to the AR device.

8. The method of claim 7 , wherein the messaging application is configured to apply one or more machine learning models to one or more images depicting the user that have been captured by the mobile device to generate the interaction data, wherein the user is using the AR device.

9. The method of claim 1 , wherein the first AR object comprises an avatar representing the user, further comprising:

determining that the movement of the user comprises a fitness activity; and

animating the avatar to represent the fitness activity.

10. The method of claim 9 , further comprising:

determining, based on the interaction data, that a limb of a user is being moved in a determinable direction; and

in response to determining that the limb of the user is being moved in the determinable direction, animating a limb of the avatar to move along the determinable direction, the limb of the avatar corresponding to the limb of the user.

11. The method of claim 1 , further comprising:

determining, based on the interaction data, that a specified limb of the user has been moved in a particular direction until reaching a certain threshold and held in that position for a threshold period of time, the certain threshold representing completion of a single repetition of a fitness activity;

updating the first AR object to mimic movement of the specified limb of the user;

computing a quantity of times that the specified limb has repeated been moved in the particular direction until reaching the certain threshold and held in that position for the threshold period of time; and

updating a counter that is presented on the AR device to represent the quantity of times.

12. The method of claim 10 , further comprising:

determining, by the AR device, that movement of the limb of the avatar corresponds to a completed iteration of the fitness activity; and

in response to determining by the AR device that movement of the limb of the avatar corresponds to the completed iteration of the fitness activity, updating a counter representing a quantity of completed iterations of the fitness activity.

13. The method of claim 12 , further comprising displaying, by the AR device, a representation of the counter together with the avatar.

14. The method of claim 1 , wherein the external client device is placed on a surface, and wherein the camera of the external client device points towards the user who is using the AR device to capture the whole body of the user, wherein the external client device generates the interaction data based on one or more images captured by the camera of the external client device.

15. A system comprising:

a storage device of an augmented reality (AR) device; and

at least one processor coupled to the storage device and configured to perform operations comprising:

establishing a communication session with an external client device;

receiving a communication from the external client device indicating that a whole body of a user has been detected within view of a camera of the external client device;

in response to receiving the communication, presenting an AR notification to inform the user that the AR device is ready to control a first AR object;

causing, by the AR device, overlay of the first AR object on a real-world environment being viewed using the AR device;

receiving, by the AR device, interaction data from the external client device representing movement of the user determined by the external client device; and

in response to receiving the interaction data from the external client device, modifying the first AR object by the AR device.

16. The system of claim 15 , the operations comprising:

presenting, as the first AR object, an avatar comprising a skier going downhill on a virtual mountain;

presenting additional virtual objects animated as approaching the avatar; and

moving a body of the avatar based on the interaction data to avoid collision between the avatar and the additional virtual objects.

17. The system of claim 16 , the operations comprising:

determining that the interaction data indicates that the user has leaned towards a direction by a certain amount of degrees; and

updating an orientation of the avatar to lean towards the direction by the certain amount of degrees.

18. The system of claim 16 , the operations comprising updating a score by the AR device based on a quantity of the additional virtual objects with which the avatar has successfully avoided collision.

19. The system of claim 15 , the operations comprising:

determining, based on the interaction data, that a specified limb of the user has been moved in a particular direction until reaching a certain threshold, the certain threshold representing completion of a single repetition of a fitness activity;

updating the first AR object to mimic movement of the specified limb of the user;

computing a quantity of times that the specified limb has repeated been moved in the particular direction until reaching the certain threshold; and

updating a counter that is presented on the AR device to represent the quantity of times.

20. A non-transitory machine-readable storage medium comprising instructions that, when executed by one or more processors of a machine, cause the machine to perform operations comprising:

establishing, by one or more processors of an augmented reality (AR) device, a communication session with an external client device;

receiving a communication from the external client device indicating that a whole body of a user has been detected within view of a camera of the external client device;

in response to receiving the communication, presenting an AR notification to inform the user that the AR device is ready to control a first AR object;

causing, by the AR device, overlay of the first AR object on a real-world environment being viewed using the AR device;

receiving, by the AR device, interaction data from the external client device representing movement of the user determined by the external client device; and

in response to receiving the interaction data from the external client device, modifying the first AR object by the AR device.

Assignments (6)