Granted Patent B2

US 12,206,914 · App. 18/021,636 · Granted Jan 21, 2025

Inventors: Yilin Wang (San Jose, CA); Balineedu Adsumilli (Foster City, CA); Junjie Ke (Stanford, CA); Hossein Talebi (San Francisco, CA); Joong Yim (Mountain View, CA); Neil Birkbeck (Santa Cruz, CA); Peyman Milanfar (Menlo Park, CA); Feng Yang (Sunnyvale, CA)

Assignee: Google LLC

H04N21/23418H04N19/154H04N21/4668

Abstract

Claims (46)

1. A method for video quality assessment, the method comprising:

receiving a video content item;

extracting a plurality of frames from the video content item;

determining, using a first subnetwork of a deep neural network, a content quality indicator for each frame of the plurality of frames of the video content item, wherein the content quality indicator represents semantic-level embeddings for each frame of the plurality of frames of the video content item;

determining, using a second subnetwork of the deep neural network, a video distortion indicator for each frame of the plurality of frames of the video content item, wherein the video distortion indicator indicates a quality of the frame based on distortions contained within the frame;

determining, using a third subnetwork of the deep neural network, a compression sensitivity indicator for each frame of the plurality of frames of the video content item;

generating a quality level for each frame of the plurality of frames of the video content item that concatenates the content quality indicator, the video distortion indicator, and the compression sensitivity indicator for that frame of the video content item;

generating an overall quality level for video content item by aggregating the quality level of each frame of the plurality of frames; and

causing a video recommendation to be presented based on the overall quality level of the video content item.

2. The method of claim 1 , wherein the first subnetwork of the deep neural network further outputs predicted content labels that describe content appearing in each frame of the plurality of frames of the video content item.

3. The method of claim 1 , wherein the video distortion indicator further represents distortion-sensitive embeddings for each frame of the plurality of frames of the video content item and wherein the second subnetwork of the deep neural network further outputs detected distortion types that describe distortion detected in each frame of the plurality of frames of the video content item.

4. The method of claim 1 , wherein the compression sensitivity indicator represents compression-sensitive embedding features for each frame of the plurality of frames of the video content item and wherein the third subnetwork of the deep neural network further outputs a compression level score.

5. The method of claim 1 , wherein the overall quality level is generated using a convolutional neural network that outputs a respective quality level for each frame of the plurality of frames of the video content item and that averages the respective quality levels.

6. The method of claim 1 , wherein the video recommendation includes a recommendation to further compress the video content item based on the overall quality level.

7. The method of claim 1 , wherein the video recommendation includes a recommendation to an uploader of the video content item to modify a portion of the video content item based on the quality level associated with that frame of the video content item.

8. A system for video quality assessment, the system comprising:

a hardware processor that is configured to:

receive a video content item;

extract a plurality of frames from the video content item;

determine, using a first subnetwork of a deep neural network, a content quality indicator for each frame of the plurality of frames of the video content item, wherein the content quality indicator represents semantic-level embeddings for each frame of the plurality of frames of the video content item;

determine, using a second subnetwork of the deep neural network, a video distortion indicator for each frame of the plurality of frames of the video content item, wherein the video distortion indicator indicates a quality of the frame based on distortions contained within the frame;

determine, using a third subnetwork of the deep neural network, a compression sensitivity indicator for each frame of the plurality of frames of the video content item;

generate a quality level for each frame of the plurality of frames of the video content item that concatenates the content quality indicator, the video distortion indicator, and the compression sensitivity indicator for that frame of the video content item;

generate an overall quality level for video content item by aggregating the quality level of each frame of the plurality of frames; and

cause a video recommendation to be presented based on the overall quality level of the video content item.

9. The system of claim 8 , wherein the first subnetwork of the deep neural network further outputs predicted content labels that describe content appearing in each frame of the plurality of frames of the video content item.

10. The system of claim 8 , wherein the video distortion indicator further represents distortion-sensitive embeddings for each frame of the plurality of frames of the video content item and wherein the second subnetwork of the deep neural network further outputs detected distortion types that describe distortion detected in each frame of the plurality of frames of the video content item.

11. The system of claim 8 , wherein the compression sensitivity indicator represents compression-sensitive embedding features for each frame of the plurality of frames of the video content item and wherein the third subnetwork of the deep neural network further outputs a compression level score.

12. The system of claim 8 , wherein the overall quality level is generated using a convolutional neural network that outputs a respective quality level for each frame of the plurality of frames of the video content item and that averages the respective quality levels.

13. The system of claim 8 , wherein the video recommendation includes a recommendation to further compress the video content item based on the overall quality level.

14. The system of claim 8 , wherein the video recommendation includes a recommendation to an uploader of the video content item to modify a portion of the video content item based on the quality level associated with that frame of the video content item.

15. A non-transitory computer-readable medium containing computer executable instructions that, when executed by a processor, cause the processor to:

receive a video content item;

extract a plurality of frames from the video content item;

determine, using a first subnetwork of a deep neural network, a content quality indicator for each frame of the plurality of frames of the video content item, wherein the content quality indicator represents semantic-level embeddings for each frame of the plurality of frames of the video content item;

determine, using a second subnetwork of the deep neural network, a video distortion indicator for each frame of the plurality of frames of the video content item, wherein the video distortion indicator indicates a quality of the frame based on distortions contained within the frame;

determine, using a third subnetwork of the deep neural network, a compression sensitivity indicator for each frame of the plurality of frames of the video content item;

generate a quality level for each frame of the plurality of frames of the video content item that concatenates the content quality indicator, the video distortion indicator, and the compression sensitivity indicator for that frame of the video content item;

generate an overall quality level for video content item by aggregating the quality level of each frame of the plurality of frames; and

cause a video recommendation to be presented based on the overall quality level of the video content item.

16. The non-transitory computer-readable medium of claim 15 , wherein the first subnetwork of the deep neural network further outputs predicted content labels that describe content appearing in each frame of the plurality of frames of the video content item.

17. The non-transitory computer-readable medium of claim 15 , wherein the video distortion indicator further represents distortion-sensitive embeddings for each frame of the plurality of frames of the video content item and wherein the second subnetwork of the deep neural network further outputs detected distortion types that describe distortion detected in each frame of the plurality of frames of the video content item.

18. The non-transitory computer-readable medium of claim 15 , wherein the compression sensitivity indicator represents compression-sensitive embedding features for each frame of the plurality of frames of the video content item and wherein the third subnetwork of the deep neural network further outputs a compression level score.

19. The non-transitory computer-readable medium of claim 15 , wherein the overall quality level is generated using a convolutional neural network that outputs a respective quality level for each frame of the plurality of frames of the video content item and that averages the respective quality levels.

20. The non-transitory computer-readable medium of claim 15 , wherein the video recommendation includes a recommendation to further compress the video content item based on the overall quality level.

21. The non-transitory computer-readable medium of claim 15 , wherein the video recommendation includes a recommendation to an uploader of the video content item to modify a portion of the video content item based on the quality level associated with that frame of the video content item.

Continuity (2)

References Cited (31)