Granted Patent B2

US 12,348,469 · App. 18/670,389 · Granted Jul 1, 2025

Inventors: Fredrik Bergenlid (Mountain View, CA); Vladyslav Lysychkin (Zurich, CH); Denis Burakov (Zurich, CH); Behshad Behzadi (Mountain View, CA); Andrea Terwisscha Van Scheltinga (Zurich, CH); Quentin Lascombes De Laroussilhe (Zurich, CH); Mikhail Golikov (Mountain View, CA); Koa Metter (Mountain View, CA); Ibrahim Badr (Mountain View, CA); Zaheed Sabur (Zurich, CH)

Assignee: Google LLC

H04L51/10G06F16/44G10L15/22H04N7/15H04N21/4394H04N21/4788G10L15/005G10L15/16G10L2015/223G10L25/63

Abstract

Claims (52)

1. A computer-implemented method comprising:

receiving session video content during a video communication session between a first computing device and a second computing device;

detecting, in the session video content, a gesture performed by a user associated with the first computing device;

determining, with a trained machine-learning model and based on the gesture, that the user invoked a request for assistance, wherein the request comprises a request for media and wherein the trained machine-learning model outputs a confidence score associated with the determination;

in response to the confidence score meeting a threshold, outputting, by the trained machine-learning model, the media; and

sending a first command to at least one of the first computing device or the second computing device to display the media.

2. The method of claim 1 , wherein the first command causes the at least one of the first computing device or the second computing device to display the media in a user interface as an illustration that is overlaid atop the session video content.

3. The method of claim 2 , further comprising:

determining, with the trained machine-learning model, an emotion associated with the user during the video communication session;

wherein the illustration is selected based on the gesture and the emotion associated with the user during the video communication session.

4. The method of claim 2 , wherein the illustration is selected from a group of a heart, a birthday-related graphic, a trophy, and combinations thereof.

5. The method of claim 1 , further comprising:

performing speech-to-text translation of audio associated with the user using the video communication session to obtain text;

wherein determining that the user associated with the first computing device invoked the request for media is further based on the text.

6. The method of claim 1 , wherein the trained machine-learning model is personalized to the user by:

modifying one or more parameters of a pre-trained machine-learning model based on the user to obtain the trained machine-learning model.

7. The method of claim 1 , wherein determining that the user associated with the first computing device invoked the request for media is further based on at least one detection selected from a group of face detection, motion detection, and combinations thereof.

8. A computing device comprising:

a processor; and

a memory coupled to the processor, with instructions stored thereon that, when executed by the processor, cause the processor to perform operations comprising:

receiving session video content during a video communication session between a first computing device and a second computing device;

detecting, in the session video content, a gesture performed by a user associated with the first computing device;

determining, with a trained machine-learning model and based on the gesture, that the user invoked a request for assistance, wherein the request comprises a request for media and wherein the trained machine-learning model outputs a confidence score associated with the determination;

in response to the confidence score meeting a threshold, outputting, by the trained machine-learning model, the media; and

sending a first command to at least one of the first computing device or the second computing device to display the media.

9. The computing device of claim 8 , wherein the first command causes the at least one of the first computing device or the second computing device to display the requested media in a user interface as an illustration that is overlaid atop the session video content.

10. The computing device of claim 9 , wherein the operations further include:

determining, with the trained machine-learning model, an emotion associated with the user during the video communication session;

wherein the illustration is selected based on the gesture and the emotion associated with the user during the video communication session.

11. The computing device of claim 9 , wherein the illustration is selected from a group of a heart, a birthday-related graphic, a trophy, and combinations thereof.

12. The computing device of claim 8 , wherein the operations further include:

performing speech-to-text translation of audio associated with the user using the video communication session to obtain text;

wherein determining that the user associated with the first computing device invoked the request for media is further based on the text.

13. The computing device of claim 8 , wherein the trained machine-learning model is personalized to the user by:

modifying one or more parameters of a pre-trained machine-learning model based on the user to obtain the trained machine-learning model.

14. The computing device of claim 8 , wherein determining that the user associated with the first computing device invoked the request for media is further based on at least one detection selected from a group of face detection, motion detection, and combinations thereof.

15. A non-transitory computer-readable medium with instructions stored thereon that, when executed by one or more computers, cause the one or more computers to perform operations, the operations comprising:

receiving session video content during a video communication session between a first computing device and a second computing device;

detecting, in the session video content, a gesture performed by a user associated with the first computing device;

determining, with a trained machine-learning model and based on the gesture, that the user invoked a request for assistance, wherein the request comprises a request for media and wherein the trained machine-learning model outputs a confidence score associated with the determination;

in response to the confidence score meeting a threshold, outputting, by the trained machine-learning model, the media; and

sending a first command to at least one of the first computing device or the second computing device to display the media.

16. The non-transitory computer-readable medium of claim 15 , wherein the first command causes the at least one of the first computing device or the second computing device to display the requested media in a user interface as an illustration that is overlaid atop the session video content.

17. The non-transitory computer-readable medium of claim 16 , wherein the operations further include:

determining, with the trained machine-learning model, an emotion associated with the user during the video communication session;

wherein the illustration is selected based on the gesture and the emotion associated with the user during the video communication session.

18. The non-transitory computer-readable medium of claim 16 , wherein the illustration is selected from a group of a heart, a birthday-related graphic, a trophy, and combinations thereof.

19. The non-transitory computer-readable medium of claim 15 , wherein the operations further include:

performing speech-to-text translation of audio associated with the user using the video communication session to obtain text;

wherein determining that the user associated with the first computing device invoked the request for media is further based on the text.

20. The non-transitory computer-readable medium of claim 15 , wherein the trained machine-learning model is personalized to the user by:

modifying one or more parameters of a pre-trained machine-learning model based on the user to obtain the trained machine-learning model.

Assignments (1)

Continuity (6)

References Cited (23)