IP Library Granted Patent US 11,743,417
Granted Patent B2
US 11,743,417 · App. 17/163,442 · Granted Aug 29, 2023

Composite video with live annotation

Inventor: Shane Paul Springer (Manchester, MI)
Assignee: Zoom Video Communications, Inc.
H04N5/272G06T11/60H04N7/15G06T2207/20212G06T2207/20221
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,743,417
App. No.
17/163,442
Granted
Aug 29, 2023
Kind
B2
Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media provide for composite video with live annotations within a video communications platform. First, the system receives video content depicting imagery of a first user, with the video content having multiple video frames. The system then determines a boundary about the first user in the video frames, with the boundary having an interior portion and an exterior portion. The system provides a media background for the exterior portion, then generates a composite video for display on client devices, with the composite video depicting the media background in a first layer and the first user in a second layer. The system then receives one or more annotation inputs corresponding to the composite video. The annotation inputs can include media-based and/or messaging-based annotations. Finally, the system updates the composite video to additionally depict the annotation inputs within a third layer.

Claims (49)

1. A method, comprising:

receiving video content depicting imagery of a first user, the video content having multiple video frames;

determining a boundary about the first user in the video frames, wherein the boundary has an interior portion and an exterior portion;

providing a media background for the exterior portion;

generating, for display on one or more client devices associated with at least one of the first user or additional users, a composite video depicting the media background in a first layer and the first user from the interior portion in a second layer, wherein the second layer is overlaid on top of the first layer;

receiving one or more annotation inputs from at least one of the client devices associated with the first user, the annotation inputs corresponding to the composite video, the annotation inputs being messaging-based;

updating the composite video to additionally depict the annotation inputs within a third layer, wherein the third layer is overlaid on top of the second layer; and

providing one or more of the client devices associated with the first user with an ability to grant access rights to an additional user of the additional users to share additional presentation content to be depicted within the composite video in real time, such that additional annotations from the first user correspond to the additional presentation content.

2. The method of claim 1 , wherein the video content comprises a live video stream or broadcast, pre-recorded video, or a combination thereof.

3. The method of claim 1 , wherein the first user appears overlaid on a sub portion of the media background within the composite video.

4. The method of claim 1 , further comprising:

providing the media background to a second user; and

receiving one or more annotation inputs corresponding to the media background and comprising graphical and/or textual annotations.

5. The method of claim 1 , wherein generating the composite video is performed in real time or substantially real time upon receiving the video content and the media background, and wherein updating the composite video is performed in real time or substantially real time upon receiving the one or more annotation inputs.

6. The method of claim 1 , wherein the first user can assign one or more of the additional users into one or more subgroups, wherein each of the subgroups is transferred to a separate video conferencing room, and wherein each of the subgroups in the separate video conferencing rooms is presented with the composite video depicting a same media background within the first layer.

7. The method of claim 6 , wherein the first user is provided with an ability to switch a view on a client device of the first user to any one of composite videos from the subgroups or any combination of a plurality of the composite videos from the subgroups.

8. The method of claim 1 , wherein at least one of the annotation inputs is transparent or semi-transparent.

9. The method of claim 1 , wherein at least one of the annotation inputs is received from at least one of the client devices associated with additional users.

10. The method of claim 1 , further comprising:

receiving one or more resizing inputs corresponding to the video content; and

in response to the one or more resizing inputs, repositioning or resizing the video content in real time or substantially real time.

11. The method of claim 1 , further comprising:

receiving a selection of one or more additional pieces of video content to be included in the composite video; and

updating the composite video to depict the additional pieces of video content within one of the first, the second layer, or the third layer.

12. The method of claim 11 , wherein at least one of the additional pieces of video content depicts imagery of an additional user, material presented by an additional user, or a combination thereof.

13. The method of claim 11 , wherein the selection is received from either the first user or one of the additional users.

14. The method of claim 1 , further comprising:

periodically or continuously detecting a network bandwidth of each of the client devices associated with the additional users; and

dynamically adjusting a video quality of at least one of the first layer, the second layer, or the third layer of the composite video displayed on at least one of the client devices in real time or substantially real time based on the detected network bandwidth of the client devices.

15. The method of claim 1 , further comprising:

sending one or more reports from the additional users to the first user of one or more elements of the composite video being displayed with inferior video quality;

receiving a request from the first user to adjust a video quality of the one or more elements of the composite video; and

adjusting, in real time or substantially real time, the video quality of the one or more elements of the composite video based on the request from the first user.

16. The method of claim 1 , further comprising:

receiving, from a client device associated with one of the additional users, a request to remove the first user from the composite video; and

updating, for display on the client device associated with the additional user, the composite video such that the first user within the second layer is removed while the second layer and the third layer remain.

17. The method of claim 1 , further comprising:

receiving a selection to replace the first user within the second layer of the composite video with one or more additional users; and

updating, for display on at least one of the client devices associated with additional users, the composite video such that the imagery of the first user within the first layer is replaced with the imagery of one or more of the additional users.

18. A communication system comprising one or more processors configured to perform operations including:

receiving video content depicting imagery of a first user, the video content having multiple video frames;

determining a boundary about the first user in the video frames, wherein the boundary has an interior portion and an exterior portion;

providing a media background for the exterior portion;

generating, for display on one or more client devices associated with at least one of the first user or additional users, a composite video depicting the media background in a first layer and the first user from the interior portion in a second layer, wherein the second layer is overlaid on top of the first layer;

receiving one or more annotation inputs from at least one of the client devices associated with the first user, the annotation inputs corresponding to the composite video, and the annotation inputs being messaging-based;

updating the composite video to additionally depict the annotation inputs within a third layer, wherein the third layer is overlaid on top of the second layer; and

providing one or more of the client devices associated with the first user with an ability to grant access rights to an additional user to share additional presentation content to be depicted within the composite video in real time, such that additional annotations from the first user correspond to the additional presentation content.

19. The communication system of claim 18 , wherein the video content is received from one or more video cameras connected to a client device associated with at least one of the first user or the one or more of the client devices associated with the additional users.

20. The communication system of claim 18 , wherein the first user and the additional users are users of a video communication platform and are connected remotely within a virtual video communication room generated by the video communication platform.

Assignments (5)
CHANGE OF NAME Recorded Jan 7, 2025
From: ZOOM VIDEO COMMUNICATIONS, INC.
To: ZOOM COMMUNICATIONS, INC.
Reel/Frame 069839/0593 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Apr 17, 2023
From: SPRINGER, SHANE PAUL
To: ZOOM VIDEO COMMUNICATIONS, INC.
Reel/Frame 063339/0347 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Sep 4, 2022
From: SPRINGER, PAUL SHANE
To: ZOOM VIDEO COMMUNICATIONS, INC.
Reel/Frame 061377/0310 →
CORRECTIVE ASSIGNMENT TO CORRECT THE RECEIVING PARTY NAME PREVIOUSLY RECORDED AT REEL: 055353 FRAME: 0645. ASSIGNOR(S) HEREBY CONFIRMS THE ASSIGNMENT. Recorded Dec 14, 2021
From: SPRINGER, SHANE
To: ZOOM VIDEO COMMUNICATIONS, INC.
Reel/Frame 058964/0580 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Feb 19, 2021
From: SPRINGER, SHANE
To: ZOOM VIDEO COMMUNICATIONS, LTD.
Reel/Frame 055343/0645 →
Continuity (1)
Related Publication 20220247940A1 · Aug 4, 2022
Cited By (2)
US 12,407,538 US 12,530,165