Real-time video enhancement

1 . A method comprising:

identifying, during a video conference, a current frame of a video stream generated by a client device of a plurality of client devices participating in the video conference;

identifying, by a processing device during the video conference, an enhanced previous frame corresponding to an enhanced version of a previous frame in the video stream;

providing, by the processing device during the video conference, at least the current frame and the enhanced previous frame as input to a machine learning model;

obtaining, by the processing device during the video conference, an output of the machine learning model, the output of the machine learning model indicating an enhanced current frame corresponding to an enhanced version of the current frame;

causing, during the video conference, the current frame to be replaced, in the video stream, with the enhanced current frame; and

causing the video stream with the enhanced current frame to be provided for presentation on one or more of the plurality of client devices during the video conference.

2 . The method of claim 1 , wherein obtaining the output of the machine learning model comprises:

obtaining, using a first subnetwork of one or more layers of the machine learning model, a first set of tokens and a second set of tokens based on the current frame and the enhanced previous frame;

obtaining, using a second subnetwork of one or more layers of the machine learning model, a first set of encoded tokens and a second set of encoded tokens based on the first set of tokens and the second set of tokens; and

obtaining, using a third subnetwork of one or more layers of the machine learning model, the enhanced current frame based on the first set of encoded tokens and the second set of encoded tokens.

3 . The method of claim 2 , wherein obtaining, using the first subnetwork, a first set of tokens and a second set of tokens based on the current frame and the enhanced previous frame comprises:

generating, using a convolutional neural network of the first subnetwork, the first set of tokens containing spatial and temporal information associated with a combination of the current frame and the enhanced previous frame.

4 . The method of claim 2 , wherein obtaining, using the second subnetwork, a first set of encoded tokens and a second set of encoded tokens comprises:

generating, using a first mixer layer of the second subnetwork, the first set of encoded tokens based on encoding the first set of tokens with learned spatio-temporal features; and

generating, using a second mixer layer of the second subnetwork, the second set of encoded tokens based on encoding the second set of tokens with learned temporal features.

5 . The method of claim 2 , wherein obtaining, using the first subnetwork, a first set of tokens and a second set of tokens based on the current frame and the enhanced previous frame comprises:

generating, using a first linear layer of the first subnetwork, a first set of intermediary tokens containing temporal information associated with the current frame;

generating, using a second linear layer of the first subnetwork, a second set of intermediary tokens containing temporal information associated with the previous frame; and

combining the first set of intermediary tokens and the second set of intermediary tokens into the second set of tokens.

6 . The method of claim 2 , wherein obtaining, using the third subnetwork, the enhanced current frame based on the first set of encoded tokens and the second set of encoded tokens comprises:

combining the first set of encoded tokens and the second set of encoded tokens; and

generating, using a decoder of the third subnetwork, the enhanced current frame based on the combination of the first set of encoded tokens and the second set of encoded tokens.

7 . The method of claim 1 , wherein the machine learning model is trained to perform at least one of: deblurring, denoising, or super-resolution to correct a low quality of a frame.

8 . A system comprising:

a memory device; and

a processing device coupled to the memory device, wherein the processing device is to perform operations comprising:

identifying, during a video conference, a current frame of a video stream generated by a client device of a plurality of client devices participating in the video conference;

identifying, during the video conference, an enhanced previous frame corresponding to an enhanced version of a previous frame in the video stream;

providing, during the video conference, at least the current frame and the enhanced previous frame as input to a machine learning model;

obtaining, during the video conference, an output of the machine learning model, the output of the machine learning model indicating an enhanced current frame corresponding to an enhanced version of the current frame;

causing, during the video conference, the current frame to be replaced, in the video stream, with the enhanced current frame; and

causing the video stream with the enhanced current frame to be provided for presentation on one or more of the plurality of client devices during the video conference.

9 . The system of claim 8 , wherein obtaining the output of the machine learning model comprises:

obtaining, using a first subnetwork of one or more layers of the machine learning model, a first set of tokens and a second set of tokens based on the current frame and the enhanced previous frame;

obtaining, using a second subnetwork of one or more layers of the machine learning model, a first set of encoded tokens and a second set of encoded tokens based on the first set of tokens and the second set of tokens; and

obtaining, using a third subnetwork of one or more layers of the machine learning model, the enhanced current frame based on the first set of encoded tokens and the second set of encoded tokens.

10 . The system of claim 9 , wherein obtaining, using the first subnetwork, a first set of tokens and a second set of tokens based on the current frame and the enhanced previous frame comprises:

generating, using a convolutional neural network of the first subnetwork, the first set of tokens containing spatial and temporal information associated with a combination of the current frame and the enhanced previous frame.

11 . The system of claim 9 , wherein obtaining, using the second subnetwork, a first set of encoded tokens and a second set of encoded tokens comprises:

generating, using a first mixer layer of the second subnetwork, the first set of encoded tokens based on encoding the first set of tokens with learned spatio-temporal features; and

generating, using a second mixer layer of the second subnetwork, the second set of encoded tokens based on encoding the second set of tokens with learned temporal features.

12 . The system of claim 9 , wherein obtaining, using the first subnetwork, a first set of tokens and a second set of tokens based on the current frame and the enhanced previous frame comprises:

generating, using a first linear layer of the first subnetwork, a first set of intermediary tokens containing temporal information associated with the current frame;

generating, using a second linear layer of the first subnetwork, a second set of intermediary tokens containing temporal information associated with the previous frame; and

combining the first set of intermediary tokens and the second set of intermediary tokens into the second set of tokens.

13 . The system of claim 9 , wherein obtaining, using the third subnetwork, the enhanced current frame based on the first set of encoded tokens and the second set of encoded tokens comprises:

combining the first set of encoded tokens and the second set of encoded tokens; and

generating, using a decoder of the third subnetwork, the enhanced current frame based on the combination of the first set of encoded tokens and the second set of encoded tokens.

14 . The system of claim 8 , wherein the machine learning model is trained to perform at least one of: deblurring, denoising, or super-resolution to correct a low quality of a frame.

15 . A non-transitory computer-readable medium comprising instructions that, responsive to execution by a processing device, cause the processing device to perform operations comprising:

identifying, during a video conference, a current frame of a video stream generated by a client device of a plurality of client devices participating in the video conference;

identifying, during the video conference, an enhanced previous frame corresponding to an enhanced version of a previous frame in the video stream;

providing, during the video conference, at least the current frame and the enhanced previous frame as input to a machine learning model;

obtaining, during the video conference, an output of the machine learning model, the output of the machine learning model indicating an enhanced current frame corresponding to an enhanced version of the current frame;

causing, during the video conference, the current frame to be replaced, in the video stream, with the enhanced current frame; and

causing the video stream with the enhanced current frame to be provided for presentation on one or more of the plurality of client devices during the video conference.

16 . The non-transitory computer-readable medium of claim 15 , wherein obtaining the output of the machine learning model comprises:

obtaining, using a first subnetwork of one or more layers of the machine learning model, a first set of tokens and a second set of tokens based on the current frame and the enhanced previous frame;

obtaining, using a second subnetwork of one or more layers of the machine learning model, a first set of encoded tokens and a second set of encoded tokens based on the first set of tokens and the second set of tokens; and

obtaining, using a third subnetwork of one or more layers of the machine learning model, the enhanced current frame based on the first set of encoded tokens and the second set of encoded tokens.

17 . The non-transitory computer-readable medium of claim 16 , wherein obtaining, using the first subnetwork, a first set of tokens and a second set of tokens based on the current frame and the enhanced previous frame comprises:

generating, using a convolutional neural network of the first subnetwork, the first set of tokens containing spatial and temporal information associated with a combination of the current frame and the enhanced previous frame.

18 . The non-transitory computer-readable medium of claim 16 , wherein obtaining, using the second subnetwork, a first set of encoded tokens and a second set of encoded tokens comprises:

generating, using a first mixer layer of the second subnetwork, the first set of encoded tokens based on encoding the first set of tokens with learned spatio-temporal features; and

generating, using a second mixer layer of the second subnetwork, the second set of encoded tokens based on encoding the second set of tokens with learned temporal features.

19 . The non-transitory computer-readable medium of claim 16 , wherein obtaining, using the first subnetwork, a first set of tokens and a second set of tokens based on the current frame and the enhanced previous frame comprises:

generating, using a first linear layer of the first subnetwork, a first set of intermediary tokens containing temporal information associated with the current frame;

generating, using a second linear layer of the first subnetwork, a second set of intermediary tokens containing temporal information associated with the previous frame; and

combining the first set of intermediary tokens and the second set of intermediary tokens into the second set of tokens.

20 . The non-transitory computer-readable medium of claim 16 , wherein obtaining, using the third subnetwork, the enhanced current frame based on the first set of encoded tokens and the second set of encoded tokens comprises:

combining the first set of encoded tokens and the second set of encoded tokens; and

generating, using a decoder of the third subnetwork, the enhanced current frame based on the combination of the first set of encoded tokens and the second set of encoded tokens.