Granted Patent B2

US 12,309,404 · App. 18/349,076 · Granted May 20, 2025

Inventors: Zhenghao Chen (Sydney, AU); Roberto Gerson De Albuquerque Azevedo (Zurich, CH); Christopher Richard Schroers (Uster, CH); Yang Zhang (Dubendorf, CH); Lucas Relic (Zurich, CH)

Assignees: Disney Enterprises, Inc.; ETH Zürich (Eidgenössische Technische Hochschule Zürich)

H04N19/42H04N19/172H04N19/91

Abstract

Claims (67)

1. A system comprising:

a first component to extract temporal features from a current frame being coded and a previous frame of a video, wherein three-dimensional based joint features are determined using the temporal features and spatial features from the current frame;

a second component that uses a first transformer to receive the three-dimensional based joint features as input and fuse the spatial features from the current frame with the temporal features to generate spatio-temporal features as first output;

a third component that uses a second transformer to perform entropy coding using the first output and at least a portion of the temporal features to generate a second output, wherein the second transformer is used to fuse the spatio-temporal features with the at least a portion of the temporal features to output fused spatio-temporal features that are entropy encoded to generate the second output; and

a fourth component that uses a third transformer to reconstruct the current frame, wherein the first output is processed using the second output to generate third output, and wherein the third transformer fuses the temporal features with the third output.

2. The system of claim 1 , wherein the three-dimensional based joint features are:

joint features for a first feature from a first input and a second feature from a second input are generated, wherein the joint features include a temporal channel.

3. The system of claim 2 , wherein at least one of the first transformer, the second transformer and the third transformer:

generates a matrix of attention weights based on the joint features; and

applies the attention weights to the joint features.

4. The system of claim 3 , wherein at least one of the first transformer, the second transformer and the third transformer:

generates a query feature, a key feature, and a value feature from the joint features; and

generates the matrix of attention weights based on the key feature and the value feature; and

applies the attention weights to the value feature.

5. The system of claim 4 , wherein the attention weights are generated by:

combining the query feature and the key feature to generate a combined output; and

applying a function to the combined output to generate the matrix for the attention weights.

6. The system of claim 5 , wherein at least one of the first transformer, the second transformer and the third transformer:

applies the attention weights to the value feature to generate a product feature.

7. The system of claim 6 , wherein at least one of the first transformer, the second transformer and the third transformer:

combines the product feature with the joint features to generate updated joint features.

8. The system of claim 7 , wherein at least one of the first transformer, the second transformer and the third transformer:

applies a gating mechanism on the updated joint features to filter information from the updated joint features to generate a gated joint spatio-temporal features.

9. The system of claim 8 , wherein at least one of the first transformer, the second transformer and the third transformer:

generates a two dimensional spatio-temporal feature from the gated joint spatio-temporal features.

10. The system of claim 1 , wherein the second component:

receives spatial information from the current frame;

combines the spatial information with a first scale of the temporal features to generate first joint spatio-temporal features;

combines, using the first transformer, the first joint spatio-temporal features with a second scale of the temporal features to generate second joint spatio-temporal features; and

combines, using the first transformer, the second joint spatio-temporal features with a third scale of the temporal features to generate third joint spatio-temporal features.

11. The system of claim 10 , wherein the third joint spatio-temporal features are used to generate an encoded bitstream of the current frame.

12. The system of claim 1 , wherein the third component:

receives a first scale of the temporal features;

receives the first output from the second component;

combines the first scale of the temporal features and the first output using the second transformer to generate a spatio-temporal prior; and

entropy codes the spatio-temporal prior to generate a distribution.

13. The system of claim 12 , wherein the distribution is used to encode the first output to an encoded bitstream and decode the encoded bitstream.

14. The system of claim 1 , wherein the fourth component:

receives the first output that is processed using the second output;

combines the first output that is processed using the second output with a first scale of the temporal features to generate first joint spatio-temporal features;

combines, using the third transformer, the first joint spatio-temporal features with a second scale of the temporal features to generate second joint spatio-temporal features; and

combines, using the third transformer, the second joint spatio-temporal features with a third scale of the temporal features to generate third joint spatio-temporal features, wherein the third joint spatio-temporal features are used to reconstruct the current frame.

15. A method comprising:

extracting temporal features from a current frame being coded and a previous frame of a video, wherein three-dimensional based joint features are determined using the temporal features and spatial features from the current frame;

using a first transformer to receive the three-dimensional based joint features as input and fuse the spatial features from the current frame with the temporal features to generate spatio-temporal features as first output;

using a second transformer to perform entropy coding using the first output and at least a portion of the temporal features to generate a second output, wherein the second transformer is used to fuse the spatio-temporal features with the at least a portion of the temporal features to output fused spatio-temporal features that are entropy encoded to generate the second output; and

using a third transformer to reconstruct the current frame, wherein the first output is processed using the second output to generate a third output, and wherein the third transformer fuses the temporal features with the third output.

16. The method of claim 15 , further comprising:

receiving a first input of a first feature and a second input of a second feature; and

generating joint feature for the first feature and the second feature, wherein the joint features include a temporal channel.

17. The method of claim 16 , wherein at least one of the first transformer, the second transformer and the third transformer:

generates a matrix of attention weights based on the joint features; and

applies the attention weights to the joint features.

18. The method of claim 17 , wherein at least one of the first transformer, the second transformer and the third transformer:

generates a query feature, a key feature, and a value feature from the joint spatio features; and

generates the matrix of attention weights based on the key feature and the value feature; and

applies the attention weights to the value feature.

19. The method of claim 18 , wherein the attention weights are generated by:

combining the query feature and the key feature to generate a combined output; and

applying a function to the combined output to generate the matrix for the attention weights.

20. An apparatus comprising:

one or more computer processors; and

a computer-readable storage medium comprising instructions for controlling the one or more computer processors to be operable for:

extracting temporal features from a current frame being coded and a previous frame of a video, wherein three-dimensional based joint features are determined using the temporal features and spatial features from the current frame;

using a first transformer to receive the three-dimensional based joint features as input and fuse the spatial features from the current frame with the temporal features to generate spatio-temporal features as first output;

using a second transformer to perform entropy coding using the first output and at least a portion of the temporal features to generate a second output, wherein the second transformer is used to fuse the spatio-temporal features with the at least a portion of the temporal features to output fused spatio-temporal features that are entropy encoded to generate the second output; and

using a third transformer to reconstruct the current frame, wherein the first output is processed using the second output to generate a third output and wherein the third transformer fuses the temporal features with the third output.

Assignments (2)

Continuity (2)

References Cited (55)