IP Library Granted Patent US 11,012,718
Granted Patent B2
US 11,012,718 · App. 16/557,920 · Granted May 18, 2021

Systems and methods for generating a latent space residual

Inventors: Christopher Schroers (Zurich, CH); Joaquim Campos (Lausanne, CH); Abdelaziz Djelouah (Zurich, CH); Yuanyi Xue (Kensington, CA); Erika Varis Doggett (Los Angeles, CA); Jared McPhillen (Glendale, CA); Scott Labrozzi (Cary, NC)
Assignee: Disney Enterprises, Inc.
H04N19/91G06K9/00744G06K9/00765G06N3/08H04N19/12H04N19/124
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,012,718
App. No.
16/557,920
Granted
May 18, 2021
Kind
B2
Abstract

Systems and methods are disclosed for generating a latent space residual. A computer-implemented method may use a computer system that includes non-transient electronic storage, a graphical user interface, and one or more physical computer processors. The computer-implemented method may include: obtaining a target frame, obtaining a reconstructed frame, encoding the target frame into a latent space to generate a latent space target frame, encoding the reconstructed frame into the latent space to generate a latent space reconstructed frame, and generating a latent space residual based on the latent space target frame and the latent space reconstructed frame.

Claims (48)

1. A computer-implemented method comprising:

obtaining, from a non-transient electronic storage, a target frame;

obtaining, from the non-transient electronic storage, a reconstructed frame, wherein the reconstructed frame is based on surrounding reference frames;

encoding, with a physical computer processor, the target frame into a latent space to generate a latent space target frame;

encoding, with the physical computer processor, the reconstructed frame into the latent space to generate a latent space reconstructed frame; and

generating, with the physical computer processor, a latent space residual based on the latent space target frame and the latent space reconstructed frame.

2. The computer-implemented method of claim 1 , further comprising decoding, with the physical computer processor, the latent space residual and the latent space reconstructed frame to generate a decoded target frame.

3. The computer-implemented method of claim 1 , wherein the reconstructed frame is generated by:

obtaining, from the non-transient electronic storage, the surrounding reference frames;

encoding, with the physical computer processor, the surrounding reference frames;

decoding, with the physical computer processor, the surrounding reference frames to generate one or more decoded reference frames; and

predicting, with the physical computer processor, the reconstructed frame based on the one or more decoded reference frames.

4. The computer-implemented method of claim 1 , wherein encoding the target frame and the reconstructed frame maps the target frame and the reconstructed frame from an image space to the latent space.

5. The computer-implemented method of claim 1 , wherein the latent space residual and the latent space reconstructed frame are quantized in the latent space.

6. The computer-implemented method of claim 1 , wherein the latent space residual and the latent space reconstructed frame are entropy coded.

7. A computer-implemented method comprising:

obtaining, from a non-transient electronic storage, a target frame;

obtaining, from the non-transient electronic storage, one or more reference frames surrounding the target frame;

obtaining, from the non-transient electronic storage, an encoder and a decoder;

applying, with a physical computer processor, the one or more reference frames to the decoder to generate one or more decoded reference frames;

predicting, with the physical computer processor, a reconstructed frame corresponding to the target frame based on the one or more decoded reference frames,

applying, with the physical computer processor, the target frame to the encoder to generate a latent space target frame;

applying, with the physical computer processor, the reconstructed frame to the encoder to generate a latent space reconstructed frame; and

generating, with the physical computer processor, a latent space residual based on the latent space target frame and the latent space reconstructed frame.

8. The computer-implemented method of claim 7 , further comprising applying, with the physical computer processor, the latent space residual and the latent space reconstructed frame to the decoder to generate a decoded target frame.

9. The computer-implemented method of claim 7 , wherein the encoder maps an image space to the latent space.

10. The computer-implemented method of claim 7 , wherein the decoder maps the latent space to an image space.

11. The computer-implemented method of claim 7 , wherein obtaining the encoder and the decoder comprises obtaining, from the non-transient electronic storage, an image transformative model, wherein the image transformative model comprises the encoder and the decoder, and wherein the image transformative model is based on a neural network.

12. The computer-implemented method of claim 7 , wherein the latent space residual and the latent space reconstructed frame are quantized in the latent space.

13. The computer-implemented method of claim 7 , wherein the latent space residual and the latent space reconstructed frame are entropy coded.

14. A system for generating a latent space residual, the system comprising:

non-transient electronic storage;

a physical computer processor configured by machine-readable instructions to:

obtain a target frame;

obtain a reconstructed frame, wherein the reconstructed frame is based on surrounding reference frames;

encode the target frame into a latent space to generate a latent space target frame;

encode the reconstructed frame into the latent space to generate a latent space reconstructed frame; and

generate a latent space residual based on the latent space target frame and the latent space reconstructed frame.

15. The system of claim 14 , wherein the physical computer processor is further configured by machine-readable instructions to decode the latent space residual and the latent space reconstructed frame to generate a decoded target frame.

16. The system of claim 15 , wherein the physical computer processor is further configured by machine-readable instructions to display, via a graphical user interface, the decoded target frame.

17. The system of claim 14 , wherein the physical computer processor is further configured by machine-readable instructions to:

obtain the surrounding reference frames;

encode the surrounding reference frames;

decode the surrounding frames to generate one or more decoded reference frames; and

predict the reconstructed frame based on the one or more decoded reference frames.

18. The system of claim 14 , wherein encoding the target frame and the reconstructed frame maps the target frame and the reconstructed frame from an image space to the latent space.

19. The system of claim 14 , wherein the latent space residual and the latent space reconstructed frame are quantized in the latent space.

20. The system of claim 14 , wherein the latent space residual and the latent space reconstructed frame are entropy coded.

Assignments (3)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Oct 1, 2019
From: SCHROERS, CHRISTOPHER; CAMPOS, JOAQUIM; DJELOUAH, ABDELAZIZ
To: THE WALT DISNEY COMPANY (SWITZERLAND) GMBH
Reel/Frame 050590/0990 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Oct 1, 2019
From: DOGGETT, ERIKA VARIS; MCPHILLEN, JARED; LABROZZI, SCOTT; XUE, YAUNYI
To: DISNEY ENTERPRISES, INC.
Reel/Frame 050591/0001 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Oct 1, 2019
From: THE WALT DISNEY COMPANY (SWITZERLAND) GMBH
To: DISNEY ENTERPRISES, INC.
Reel/Frame 050598/0694 →
Continuity (1)
Related Publication 20210067808A1 · Mar 4, 2021