IP Library › Granted Patent US 12,425,556
Granted Patent B2
US 12,425,556 · App. 17/727,970 · Granted Sep 23, 2025

Learning-based light field compression for tensor display

Inventor: Zhu Li (Overland Park, KS)
Assignee: Adeia Guides Inc.
H04N13/122H04N13/161H04N13/388
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,425,556
App. No.
17/727,970
Filed
Apr 25, 2022
Granted
Sep 23, 2025
Kind
B2
Art Unit
2482
USPC
348/43
Abstract

Systems, methods and apparatuses are described herein for training a machine learning model to accept as input synthetic aperture image (SAI) training data for a three-dimensional (3D) display, the 3D display comprising a plurality of layers. The machine learning model may be trained to output respective pixel representations of the SAI training data for each of the plurality of layers of the 3D display. The provided systems, methods and apparatuses may access image data, input the image data to the trained machine learning model, and determine, using the trained machine learning model, respective pixel representations of the input image data for each of the plurality of layers of the 3D display. The provided systems, methods and apparatuses may encode the respective pixel representations of the input image data, and transmit, for display at the 3D display, the encoded respective pixel representations of the input image data.

Claims (60)

1. A computer-implemented method comprising:

training a machine learning model to:

accept as input synthetic aperture image (SAI) training data for a three-dimensional (3D) display, the 3D display comprising a plurality of layers; and

output respective pixel representations of the SAI training data for each of the plurality of layers of the 3D display;

inputting image data to the trained machine learning model;

determining, using the trained machine learning model, respective pixel representations of the input image data for each of the plurality of layers of the 3D display;

encoding, at one or more servers, the respective pixel representations of the input image data, wherein the encoding comprises compressing the respective pixel representations of the input image data; and

transmitting, by the one or more servers to the 3D display over a communications network, the encoded respective pixel representations of the input image data, wherein the 3D display decodes the encoded respective pixel representations of the input image data and displays content based on the decoding.

2. The method of claim 1 , wherein:

the 3D display is a light field (LF) tensor display, and

the SAI training data comprises LF information and represents respective view angles of a plurality of view angles of a frame of a media asset.

3. The method of claim 1 , wherein training the machine learning model further comprises:

obtaining, using a least-squares solver, an initial estimate for the respective pixel representations for each of the plurality of layers;

determining a loss function based on the initial estimate; and

adjusting one or more parameters of the machine learning model to minimize the loss function.

4. The method of claim 1 , wherein the machine learning model is a deep learning deformable SAI feature embedding network comprising:

a deformable feature extraction block comprising a deformable convolution layer and a deformable layer; and

a residual learning block.

5. The method of claim 4 , wherein training the machine learning model further comprises:

causing the machine learning model to learn, using the deformable convolution layer and for each feature map of a plurality of feature maps representing characteristics of respective pixels of the SAI training data, filter weights for a filter and an offset mask for the filter.

6. The method of claim 5 , wherein the learned offset mask enables flexible selection of input feature map pixels.

7. The method of claim 5 , wherein the filter is configured to be slid around, and convolved with, pixels of the SAI training data at a plurality of sampling positions, and the offset mask is configured to deform the plurality of sampling positions.

8. The method of claim 7 , wherein the offset mask is fractional, and the deformable convolution layer is configured to perform bilinear interpolation to estimate pixel values of the deformed sampling positions.

9. The method of claim 1 , wherein:

a second layer of the 3D display is disposed between a first layer of the 3D display and a third layer of the 3D display, and is spaced apart from the first layer and the third layer;

the first layer is disposed between a backlight of the 3D display and the second layer, and is spaced apart from the backlight and the second layer; and

a distance between the third layer and the backlight is greater than a distance between the second layer and the backlight, and the distance between the second layer and the backlight is greater than a distance between the first layer and the backlight.

10. The method of claim 1 , wherein the encoding further comprises:

applying versatile vide coding (VVC) intra coding to the respective pixel representations of the input image data determined by the trained machine learning model.

11. A computer-implemented system comprising:

input/output (I/O) circuitry; and

control circuitry configured to:

train a machine learning model to:

accept as input synthetic aperture image (SAI) training data for a three-dimensional (3D) display, the 3D display comprising a plurality of layers; and

output respective pixel representations of the SAI training data for each of the plurality of layers of the 3D display;

input image data to the trained machine learning model, wherein the image data is accessed via the I/O circuitry;

determine, using the trained machine learning model, respective pixel representations of the input image data for each of the plurality of layers of the 3D display;

encode, at one or more servers, the respective pixel representations of the input image data, wherein the encoding comprises compressing the respective pixel representations of the input image data; and

transmit, by the one or more servers to the 3D display over a communications network, the encoded respective pixel representations of the input image data, wherein the 3D display decodes the encoded respective pixel representations of the input image data and displays content based on the decoding.

12. The system of claim 11 , wherein:

the 3D display is a light field (LF) tensor display, and

the SAI training data comprises LF information and represents respective view angles of a plurality of view angles of a frame of a media asset.

13. The system of claim 11 , wherein the control circuitry is configured to train the machine learning model by:

obtaining, using a least-squares solver, an initial estimate for the respective pixel representations for each of the plurality of layers;

determining a loss function based on the initial estimate; and

adjusting one or more parameters of the machine learning model to minimize the loss function.

14. The system of claim 11 , wherein the machine learning model is a deep learning deformable SAI feature embedding network comprising:

a deformable feature extraction block comprising a deformable convolution layer and a deformable layer; and

a residual learning block.

15. The system of claim 14 , wherein the control circuitry is configured to train the machine learning model further by:

causing the machine learning model to learn, using the deformable convolution layer and for each feature map of a plurality of feature maps representing characteristics of respective pixels of the SAI training data, filter weights for a filter and an offset mask for the filter.

16. The system of claim 15 , wherein the learned offset mask enables flexible selection of input feature map pixels.

17. The system of claim 15 , wherein the control circuitry is configured to cause the filter to be slid around, and convolved with, pixels of the SAI training data at a plurality of sampling positions, and the control circuitry is configured to cause the offset mask to deform the plurality of sampling positions.

18. The system of claim 17 , wherein the offset mask is fractional, and the control circuitry is configured to cause the deformable convolution layer to perform bilinear interpolation to estimate pixel values of the deformed sampling positions.

19. The system of claim 11 , wherein:

a second layer of the 3D display is disposed between a first layer of the 3D display and a third layer of the 3D display, and is spaced apart from the first layer and the third layer,

the first layer is disposed between a backlight of the 3D display and the second layer, and is spaced apart from the backlight and the second layer, and

a distance between the third layer and the backlight is greater than a distance between the second layer and the backlight, and the distance between the second layer and the backlight is greater than a distance between the first layer and the backlight.

20. The system of claim 11 , wherein the encoding further comprises:

applying versatile vide coding (VVC) intra coding to the respective pixel representations of the input image data determined by the trained machine learning model.

Assignments (3)
CHANGE OF NAME Recorded Oct 4, 2024
From: ROVI GUIDES, INC.
To: ADEIA GUIDES INC.
Reel/Frame 069113/0406 →
SECURITY INTEREST Recorded May 3, 2023
From: ADEIA GUIDES INC.; ADEIA IMAGING LLC; ADEIA MEDIA HOLDINGS LLC; ADEIA MEDIA SOLUTIONS INC.; ADEIA SEMICONDUCTOR ADVANCED TECHNOLOGIES INC.; ADEIA SEMICONDUCTOR BONDING TECHNOLOGIES INC.; ADEIA SEMICONDUCTOR INC.; ADEIA SEMICONDUCTOR SOLUTIONS LLC; ADEIA SEMICONDUCTOR TECHNOLOGIES LLC; ADEIA SOLUTIONS LLC
To: BANK OF AMERICA, N.A., AS COLLATERAL AGENT
Reel/Frame 063529/0272 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jun 29, 2022
From: LI, ZHU
To: ROVI GUIDES, INC.
Reel/Frame 060354/0570 →
Continuity (1)
Related Publication 20230344974A1 · Oct 26, 2023
References Cited (11)
US 20230106939A1 · Yuan · 2023 [cited by examiner]
Maruyama et al. K. Maruyama, Y. Inagaki, K. Takahashi, T. Fujii and H. Nagahara, “A 3-D Display Pipeline from Coded-Aperture Camera to Tensor Light-Field Display Through CNN,” 2019 IEEE International Conference on Image… [cited by examiner]
Y. Wang et al., “Light Field Image Super-Resolution Using Deformable Convolution,” in IEEE Transactions on Image Processing, vol. 30, pp. 1057-1071, 2021, doi: 10.1109/TIP.2020.3042059 (Year: 2021). [cited by examiner]
Bross et al., “Developments in International Video Coding Standardization After AVC, With an Overview of Versatile Video Coding (VVC)”, Proceedings of the IEEE, 109(9):1463-1493 (2021). [cited by applicant]
Hedayati et al., “Light Field Compression by Residual CNN Assisted JPEG”, IJCNN, pp. 1-13 (2021). [cited by applicant]
Li et al., “Light Field and Plenoptic Point Cloud Compression,” Springer: Handbook for Dynamic Data Driven Applications Systems, pp. 199-226 (2021). [cited by applicant]
Li et al., “Pseudo Sequence Based 2-D Hierarchical Coding Structure for Light-Field Image Compression”, IEEE Journal of Selected Topics in Signal Processing, 11(7):1107-1119 (2017). [cited by applicant]
Li et al., “Quadtree-based Coding Framework for High Density Camera Array Based Light Field Image,” IEEE Trans on Circuits and Systems for Video Technology, pp. 1-16 (2019). [cited by applicant]
Wang et al., “Salience Guided Depth Calibration for Perceptually Optimized Light Field 3D Display,” Proc. CVPR, 2031-2040 (2018). [cited by applicant]
Wetzstein et al., “Tensor Displays: Compressive Light Field Synthesis using Multilayer Displays with Directional Backlighting,” ACM Trans. Graph. 31(4) 80:1-80.11 (2012). [cited by applicant]
Zhao et al., “Light Field Image Compression Based on Deep Learning,” Proc of IEEE ICME, pp. 1-6 (2018). [cited by applicant]