IP Library Granted Patent US 11,544,606
Granted Patent B2
US 11,544,606 · App. 16/254,475 · Granted Jan 3, 2023

Machine learning based video compression

Inventors: Stephan Marcel Mandt (Santa Monica, CA); Christopher Schoers (Zurich, CH); Jun Han (Hanover, NH); Salvator D. Lombardo (Glendale, CA)
Assignee: Disney Enterprises, Inc.
G06N7/005G06N3/04G06N3/08G06N20/00H04N19/42
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,544,606
App. No.
16/254,475
Filed
Jan 22, 2019
Granted
Jan 3, 2023
Kind
B2
Art Unit
2123
USPC
706/15
Abstract

Systems and methods for compressing target content are disclosed. In one embodiment, a system may include non-transient electronic storage and one or more physical computer processors. The one or more physical computer processors may be configured by machine-readable instructions to obtain the target content comprising one or more frames, wherein a given frame comprises one or more features. The one or more physical computer processors may be configured by machine-readable instructions to obtain a conditioned network. The one or more physical computer processors may be configured by machine-readable instructions to generate decoded target content by applying the conditioned network to the target content.

Claims (62)

1. A system configured for compressing target content, the system comprising:

non-transient electronic storage;

one or more physical computer processors configured by machine-readable instructions to:

obtain, from the non-transient electronic storage, the target content comprising one or more frames, wherein a given frame comprises one or more features;

obtain, from the non-transient electronic storage, a conditioned network, the conditioned network having been trained by training an initial network using training content, wherein the conditioned network comprises one or more encoders, one or more quantizers, and one or more decoders, and wherein the training content comprises one or more training frames, and wherein a given training frame comprises one or more training features;

apply, with the one or more physical computer processors, the conditioned network to the target content to generate a latent space of the target content comprising one or more local variables, one or more global variables, and a plurality of distributions corresponding to the latent space; and

quantize the one or more local variables and the one or more global variables based on the plurality of distributions corresponding to the latent space to generate encoded target content.

2. The system of claim 1 , wherein the one or more local variables are based on the one or more features in the given frame, wherein the one or more global variables are based on one or more features common to a plurality of frames of the target content, and wherein the plurality of distributions indicate a likelihood of values for the one or more local variables and the one or more global variables.

3. The system of claim 1 , wherein the one or more physical computer processors are further configured by machine-readable instructions to:

apply, with the one or more physical computer processors, a plurality of convolutional layers to the target content to generate convolved target content;

apply, with the one or more physical computer processors, a global model to the convolved target content to generate the one or more global variables; and

apply, with the one or more physical computer processors, a multilayer perceptron model to the convolved target content to generate the one or more local variables.

4. The system of claim 3 , wherein the global model comprises one or more of a long short-term memory model or a Kalman filter.

5. The system of claim 1 , wherein applying the conditioned network further comprises:

decoding, with the one or more physical computer processors, the encoded target content.

6. The system of claim 5 , wherein decoding the encoded target content comprises:

entropy decoding, with the one or more physical computer processors, the encoded target content;

combining, with the one or more physical computer processors, the entropy decoded target content with a multilayer perceptron model; and

applying, with the one or more physical computer processors, a plurality of deconvolutions to a combination of the entropy decoded target content with the multilayer perceptron model.

7. The system of claim 1 , wherein the plurality of distributions corresponding to the latent space comprise noise, and are centered around means of the one or more global variables and the one or more local variables.

8. A computer-implemented method for training an initial network to simultaneously learn how to refine a latent space using training content and how to refine a plurality of distributions of the latent space using the training content, the method being implemented in a computer system that comprises non-transient electronic storage and one or more physical computer processors, comprising:

obtaining, from the non-transient electronic storage, training content comprising one or more training frames, wherein a given training frame comprises one or more training features;

obtaining, from the non-transient electronic storage, the initial network, the initial network comprising one or more encoders, one or more quantizers, and one or more decoders; and

generating, with the one or more physical computer processors, a conditioned network by training the initial network using the training content, the conditioned network comprising the one or more encoders, the one or more quantizers, and the one or more decoders, wherein the conditioned network is trained to receive target content and generate encoded target content comprising a quantized latent space that includes one or more quantized local variables and one or more quantized global variables.

9. The computer-implemented method of claim 8 , further comprising:

obtaining, from the non-transient electronic storage, the target content comprising one or more frames, wherein a given frame comprises one or more features;

encoding, with the one or more physical computer processors, the target content to generate one or more local variables and one or more global variables using the conditioned network;

generating, with the one or more physical computer processors, the latent space using the conditioned network, the latent space comprising the one or more local variables and the one or more global variables, wherein the one or more local variables are based on the one or more features in the given frame, and wherein the one or more global variables are based on one or more features common to a plurality of frames of the target content;

generating, with the one or more physical computer processors, the plurality of distributions corresponding to the latent space using the conditioned network, wherein the plurality of distributions indicate likelihoods of values for the one or more local variables and the one or more global variables; and

quantizing, with the one or more physical computer processors, the one or more local variables and the one or more global variables based on the plurality of distributions using the conditioned network.

10. The computer-implemented method of claim 9 , further comprising:

encoding, with the one or more physical computer processors, the one or more quantized local variables and the one or more quantized global variables; and

decoding, with the one or more physical computer processors, the one or more encoded quantized local variables and the one or more encoded quantized global variables.

11. The computer-implemented method of claim 10 , wherein decoding the one or more encoded quantized local variables and the one or more encoded quantized global variables comprises:

entropy decoding, with the one or more physical computer processors, the one or more encoded quantized local variables and the one or more encoded quantized global variables;

combining, with the one or more physical computer processors, the one or more entropy decoded quantized local variables and the one or more entropy decoded quantized global variables with a multilayer perceptron model; and

applying, with the one or more physical computer processors, a plurality of deconvolutions to a combination of the one or more entropy decoded quantized local variables and the one or more entropy decoded quantized global variables with the multilayer perceptron model.

12. The computer-implemented method of claim 9 , wherein the plurality of distributions corresponding to the latent space comprise noise, and are centered around means of the one or more global variables and the one or more local variables.

13. The computer-implemented method of claim 9 , wherein encoding the target content comprises:

applying, with the one or more physical computer processors, a plurality of convolutional layers to the target content to generate convolved target content;

applying, with the one or more physical computer processors, a long short-term memory model to the convolved target content to generate the one or more global variables; and

applying, with the one or more physical computer processors, a multilayer perceptron model to the convolved target content to generate the one or more local variables.

14. A computer-implemented method for compressing target content, the method being implemented in a computer system that comprises non-transient electronic storage and one or more physical computer processors, comprising:

obtaining, from the non-transient electronic storage, the target content comprising one or more frames, wherein a given frame comprises one or more features;

encoding, with the one or more physical computer processors, the target content to generate one or more local variables and one or more global variables; and

generating, with the one or more physical computer processors, a latent space, the latent space comprising the one or more local variables and the one or more global variables, wherein the one or more local variables are based on the one or more features in the given frame, and wherein the one or more global variables are based on one or more features common to a plurality of frames of the target content;

generating, with the one or more physical computer processors, a plurality of distributions corresponding to the latent space; and

quantizing the one or more local variables and the one or more global variables based on the plurality of distributions corresponding to the latent space.

15. The computer-implemented method of claim 14 , wherein the plurality of distributions indicate a likelihood of values for the one or more local variables and the one or more global variables.

16. The computer-implemented method of claim 15 , further comprising:

encoding, with the one or more physical computer processors, a quantized latent space that includes the one or more quantized local variables and the one or more quantized global variables; and

decoding, with the one or more physical computer processors, the encoded quantized latent space.

17. The computer-implemented method of claim 16 , wherein decoding the encoded quantized latent space comprises:

entropy decoding, with the one or more physical computer processors, the encoded latent space;

combining, with the one or more physical computer processors, the entropy decoded latent space with a multilayer perceptron model; and

applying, with the one or more physical computer processors, a plurality of deconvolutions to a combination of the entropy decoded latent space with the multilayer perceptron model.

18. The computer-implemented method of claim 14 , wherein encoding the target content comprises:

applying, with the one or more physical computer processors, a plurality of convolutional layers to the target content to generate convolved target content;

applying, with the one or more physical computer processors, a long short-term memory model to the convolved target content to generate the one or more global variables; and

applying, with the one or more physical computer processors, a multilayer perceptron model to the convolved target content to generate the one or more local variables.

19. The computer-implemented method of claim 14 , wherein the plurality of distributions corresponding to the latent space are centered around means of the one or more global variables and the one or more local variables, and wherein random noise is added to the plurality of distributions.

20. The computer-implemented method of claim 14 , wherein the latent space comprises a global density model corresponding to the one or more global variables and a local density model corresponding to the one or more local variables.

Assignments (4)
CORRECTIVE ASSIGNMENT TO CORRECT THE FIRST INVENTOR'S NAME PREVIOUSLY RECORDED AT REEL: 048314 FRAME: 0671. ASSIGNOR(S) HEREBY CONFIRMS THE ASSIGNMENT. Recorded Feb 10, 2023
From: MANDT, STEPHAN MARCEL; LOMBARDO, SALVATOR D.; HAN, JUN
To: DISNEY ENTERPRISES, INC.
Reel/Frame 062702/0391 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Feb 12, 2019
From: SCHROERS, CHRISTOPHER
To: THE WALT DISNEY COMPANY (SWITZERLAND) GMBH
Reel/Frame 049921/0076 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Feb 12, 2019
From: THE WALT DISNEY COMPANY (SWITZERLAND) GMBH
To: DISNEY ENTERPRISES, INC.
Reel/Frame 049921/0126 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Feb 12, 2019
From: MANDT, STEPHEN MARCEL; LOMBARDO, SALVATOR D.; HAN, JUN
To: DISNEY ENTERPRISES, INC.
Reel/Frame 048314/0671 →
Continuity (2)
Provisional Application 62731672 · Sep 14, 2018
Related Publication 20200090069A1 · Mar 19, 2020
Cited By (2)
US 12,267,089 US 12,647,610