Adaptive teleconferencing experiences using generative image models

1 . A computer-implemented method comprising:

receiving a first video signal captured by a first camera of a first computing device during a teleconference, the first video signal depicting a first user and a first background of the first user;

receiving a second video signal captured by a second camera of a second computing device during the teleconference, the second video signal depicting a second user and a second background of the second user;

inputting at least a first mask for the first background and a second mask for the second background to a generative image model and instructing the generative image model to inpaint around the first background and the second background;

receiving an inpainted image from the generative image model, the generative image model preserving the first background based on the first mask and the second background based on the second mask when generating the inpainted image;

compositing the first user and the second user on the inpainted image to obtain a composited inpainted image; and

outputting the composited inpainted image.

2 . The computer-implemented method of claim 1 , wherein the outputting comprises transmitting the composited inpainted image over a network during the teleconference to the first computing device, the second computing device, or another computing device.

3 . The computer-implemented method of claim 1 , further comprising:

segmenting the first user from the first video signal and the second user from the second video signal to perform the compositing.

4 . The computer-implemented method of claim 1 , further comprising:

receiving user input specifying an extent of background preservation; and

determining respective sizes for the first mask and the second mask based on the user input.

5 . The computer-implemented method of claim 1 , further comprising:

receiving a base prompt relating to the teleconference from the first user or the second user; and

conditioning the generative image model based at least on the base prompt.

6 . The computer-implemented method of claim 5 , wherein the conditioning comprises:

inputting at least part of the base prompt to a generative language model to obtain an expanded prompt; and

inputting the expanded prompt to the generative image model.

7 . The computer-implemented method of claim 1 , further comprising:

receiving user input identifying a selected area of the composited inpainted image and a description of an object to place in the selected area; and

instructing the generative image model to place the object in the selected area.

8 . The computer-implemented method of claim 1 , wherein the compositing comprises:

performing hidden surface removal to place at least part of the first user or the second user behind an object in the composited inpainted image.

9 . The computer-implemented method of claim 1 , further comprising:

instructing the generative image model to restyle the composited inpainted image using an image-to-image technique based at least on a restyling prompt.

10 . A system comprising:

a processor; and

a storage medium storing instructions which, when executed by the processor, cause the system to:

receive a first video signal captured by a first camera of a first computing device during a teleconference, the first video signal depicting a first user and a first background of the first user;

receive a second video signal captured by a second camera of a second computing device during the teleconference, the second video signal depicting a second user and a second background of the second user;

input at least a first mask for the first background and a second mask for the second background to a generative image model and instruct the generative image model to inpaint around the first background and the second background;

receive an inpainted image from the generative image model, the generative image model preserving the first background based on the first mask and the second background based on the second mask when generating the inpainted image;

composite the first user and the second user on the inpainted image to obtain a composited inpainted image; and

send the composited inpainted image to at least one of the first computing device, the second computing device, or another computing device of another user.

11 . The system of claim 10 , wherein the instructions, when executed by the processor, cause the system to:

input a prompt relating to the teleconference to the generative image model.

12 . The system of claim 10 , wherein the instructions, when executed by the processor, cause the system to:

receive user input identifying a selected area of the composited inpainted image and a description of an object to place in the selected area; and

instruct the generative image model to place the object in the selected area,

the composited inpainted image having the object in the selected area.

13 . The system of claim 10 , wherein the instructions, when executed by the processor, cause the system to:

instruct the generative image model to restyle the composited inpainted image using an image-to-image technique based at least on a restyling prompt.

14 . The system of claim 13 , wherein the instructions, when executed by the processor, cause the system to:

determine the restyling prompt based on user input.

15 . A hardware computer-readable storage device storing computer-readable instructions which, when executed by a hardware processing device, cause the hardware processing device to perform acts comprising:

receiving a first video signal captured by a first camera of a first computing device during a teleconference, the first video signal depicting a first user and a first background of the first user;

receiving a second video signal captured by a second camera of a second computing device during the teleconference, the second video signal depicting a second user and a second background of the second user;

inputting at least a first mask for the first background and a second mask for the second background to a generative image model and instructing the generative image model to inpaint around the first background and the second background;

receiving an inpainted image from the generative image model, the generative image model preserving the first background based on the first mask and the second background based on the second mask when generating the inpainted image;

compositing the first user and the second user on the inpainted image to obtain a composited inpainted image; and

outputting the composited inpainted image.

16 . The hardware computer-readable storage device of claim 15 , the acts further comprising:

segmenting the first user from the first video signal and the second user from the second video signal to perform the compositing.

17 . The hardware computer-readable storage device of claim 15 , the acts further comprising:

receiving user input specifying an extent of background preservation; and

determining respective sizes for the first mask and the second mask based on the user input.

18 . The hardware computer-readable storage device of claim 15 , the acts further comprising:

receiving user input identifying a selected area of the composited inpainted image and a description of an object to place in the selected area; and

instructing the generative image model to place the object in the selected area,

wherein the composited inpainted image includes the object placed in the selected area.

19 . The hardware computer-readable storage device of claim 15 , the acts further comprising:

performing hidden surface removal to place at least part of the first user or the second user behind an object in the composited inpainted image.

20 . The hardware computer-readable storage device of claim 15 , the acts further comprising:

instructing the generative image model to restyle the composited inpainted image using an image-to-image technique based at least on a restyling prompt.