IP Library › Granted Patent US 12,749,147
Granted Patent B2
US 12,749,147 · App. 18/393,547 · Granted Sep 29, 2026

Generation super sampling

Inventor: Denis Dupeyron (Broomfield, CO)
Assignee: Sony Interactive Entertainment Inc.
G06T3/4046G06F3/0354
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,749,147
App. No.
18/393,547
Granted
Sep 29, 2026
Kind
B2
Abstract

Frame generation super sampling may include generating an image frame embedding from a first image frame in an image stream and predicting a synthetic second image frame in the image stream using the image frame embedding of the first image in the image stream. The synthetic second image is displayed frame after the first image frame in the image stream.

Claims (40)

1 . A method for frame generation super sampling, comprising:

generating a first image frame embedding from a first image frame in an image stream;

before generating a second image frame embedding from a second image in the image stream, predicting a synthetic image frame in the image stream using the first image frame embedding of the first image frame in the image stream;

displaying the synthetic image frame after the first image frame in the image stream; and

after displaying the synthetic image frame, displaying the second image frame in the image stream.

2 . The method of claim 1 , wherein generating the first image frame embedding comprises generating the first image frame embedding with user input information.

3 . The method of claim 2 , wherein the user input information includes one or more button presses of an input device.

4 . The method of claim 2 , wherein the user input information includes one or more movements of an input device.

5 . The method of claim 4 , wherein the input device is a mouse or trackball or joystick.

6 . The method of claim 4 , wherein the input device is an inertial measurement unit.

7 . The method of claim 1 , wherein generating the first image frame embedding comprises generating the first image frame embedding with motion vectors from the image stream.

8 . The method of claim 1 , wherein generating the first image frame embedding from the first image frame in the image stream comprises generating the first image frame embedding in conjunction with a previous image frame in the image stream.

9 . The method of claim 1 , wherein generating the first image frame embedding includes using a neural network trained with a machine learning algorithm.

10 . The method of claim 1 , wherein predicting the synthetic image frame in the image stream using the first image frame embedding includes using a neural network trained with a machine learning algorithm.

11 . The method of claim 1 , further comprising:

predicting another synthetic image frame using the first image frame embedding; and

displaying the other synthetic image frame after displaying the synthetic image frame in the image stream.

12 . The method of claim 11 , further comprising displaying a third image frame after displaying the other synthetic image frame.

13 . The method of claim 1 , comprising:

using the first and second image frames, determining motion vectors between a matching block in the first and second image frames; and

using the motion vectors, predicting a location of the matching block in a second synthetic image.

14 . A system for frame generation super sampling, comprising:

a processor;

a memory operatively coupled to the processor;

non-transitory processor executable instructions embodied in the memory, the non-transitory processor executable instructions when executed by the processor cause the processor to carry out a method for frame generation super sampling comprising:

generating a first image frame embedding from a first image frame in an image stream;

before generating a second image frame embedding from a second image in the image stream, predicting a synthetic image frame in the image stream using the first image frame embedding of the first image frame in the image stream;

displaying the synthetic image frame after the first image frame in the image stream; and

after displaying the synthetic image frame, displaying the second image frame in the image stream.

15 . The system of claim 14 , further comprising an input device,

wherein generating the first image frame embedding from the first image frame in the image stream uses user input information.

16 . The system of claim 15 , wherein the user input information includes one or more button presses of the input device.

17 . The system of claim 16 , wherein the user input information includes one or more movements of the input device.

18 . The system of claim 17 , wherein the input device is a mouse or trackball or joystick.

19 . The system of claim 17 , wherein the input device is an inertial measurement unit.

20 . A non-transitory computer readable medium having computer executable instructions embedded thereon, the computer executable instructions when executed by a computer cause the computer to implement a method for frame generation super sampling comprising:

generating a first image frame embedding from a first image frame in an image stream;

before generating a second image frame embedding from a second image in the image stream, predicting a synthetic image frame in the image stream using the first image frame embedding of the first image frame in the image stream;

displaying the synthetic image frame after the first image frame in the image stream; and

after displaying the synthetic image frame, displaying the second image frame in the image stream.

Assignments (2)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Mar 31, 2025
From: DUPEYRON, DENIS
To: SONY INTERACTIVE ENTERTAINMENT INC.
Reel/Frame 070676/0121 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Dec 21, 2023
From: DUPEYRON, DENIS
To: SONY INTERACTIVE ENTERTAINMENT INC.
Reel/Frame 065937/0184 →
Continuity (1)
Related Publication 20250209568A1 · Jun 26, 2025
References Cited (119)
US 5231484A · Gonzales et al. · 1993 [cited by applicant]
US 5377051A · Lane et al. · 1994 [cited by applicant]
US 6463101B1 · Koto · 2002 [cited by applicant]
US 6700932B2 · Shen et al. · 2004 [cited by applicant]
US 6700935B2 · Lee · 2004 [cited by applicant]
US 7145946B2 · Lee · 2006 [cited by applicant]
US 7321621B2 · Popescu et al. · 2008 [cited by applicant]
US 7372903B1 · Lee et al. · 2008 [cited by applicant]
US 7426296B2 · Lee et al. · 2008 [cited by applicant]
US 7697608B2 · Lee · 2010 [cited by applicant]
US 7697783B2 · Lee et al. · 2010 [cited by applicant]
US 7848428B2 · Chin · 2010 [cited by applicant]
US 8027384B2 · Lee · 2011 [cited by applicant]
US 8032520B2 · Dipper et al. · 2011 [cited by applicant]
US 8213518B1 · Wang et al. · 2012 [cited by applicant]
US 8218627B2 · Lee · 2012 [cited by applicant]
US 8218640B2 · Wang · 2012 [cited by applicant]
US 8218641B2 · Wang · 2012 [cited by applicant]
US 8345750B2 · Lee · 2013 [cited by applicant]
US 8711933B2 · Lee · 2014 [cited by applicant]
US 8737485B2 · Zhang et al. · 2014 [cited by applicant]
US 8879623B2 · Lee · 2014 [cited by applicant]
US 8913664B2 · Lee · 2014 [cited by applicant]
US 9386317B2 · Lee · 2016 [cited by applicant]
US 9872018B2 · Lee · 2018 [cited by applicant]
US 9904866B1 · Noble · 2018 [cited by applicant]
US 10178390B2 · Lee · 2019 [cited by applicant]
US 10210662B2 · Holzer · 2019 [cited by examiner]
US 10248663B1 · Keisler et al. · 2019 [cited by applicant]
US 10419760B2 · Lee · 2019 [cited by applicant]
US 11321555B2 · Cower · 2022 [cited by examiner]
US 11398060B2 · Takashima · 2022 [cited by examiner]
US 11501489B2 · Cao · 2022 [cited by examiner]
US 12266144B2 · Mustikovela · 2025 [cited by examiner]
US 20030156198A1 · Lee · 2003 [cited by applicant]
US 20040086193A1 · Kameyama et al. · 2004 [cited by applicant]
US 20040146212A1 · Kadono et al. · 2004 [cited by applicant]
US 20050147375A1 · Kadono · 2005 [cited by applicant]
US 20050169369A1 · Lee · 2005 [cited by applicant]
US 20050169370A1 · Lee · 2005 [cited by applicant]
US 20050169371A1 · Lee et al. · 2005 [cited by applicant]
US 20050207643A1 · Lee et al. · 2005 [cited by applicant]
US 20050265461A1 · Raveendran · 2005 [cited by applicant]
US 20050281334A1 · Walker et al. · 2005 [cited by applicant]
US 20070025621A1 · Lee et al. · 2007 [cited by applicant]
US 20070274396A1 · Zhang et al. · 2007 [cited by applicant]
US 20070297505A1 · Fidler et al. · 2007 [cited by applicant]
US 20080025397A1 · Zhao et al. · 2008 [cited by applicant]
US 20080049844A1 · Liu et al. · 2008 [cited by applicant]
US 20080181311A1 · Zhang et al. · 2008 [cited by applicant]
US 20090003441A1 · Sekiguchi et al. · 2009 [cited by applicant]
US 20090003447A1 · Christoffersen et al. · 2009 [cited by applicant]
US 20090003448A1 · Sekiguchi et al. · 2009 [cited by applicant]
US 20100150227A1 · Lee · 2010 [cited by applicant]
US 20100150228A1 · Lee · 2010 [cited by applicant]
US 20110051809A1 · Lee · 2011 [cited by applicant]
US 20110122942A1 · Kudana et al. · 2011 [cited by applicant]
US 20120033730A1 · Lee · 2012 [cited by applicant]
US 20130034169A1 · Sadafale et al. · 2013 [cited by applicant]
US 20130072299A1 · Lee · 2013 [cited by applicant]
US 20140161172A1 · Wang et al. · 2014 [cited by applicant]
US 20140233640A1 · Lee · 2014 [cited by applicant]
US 20150016513A1 · Lee · 2015 [cited by applicant]
US 20150256736A1 · Fukuhara · 2015 [cited by applicant]
US 20160088299A1 · Lee · 2016 [cited by applicant]
US 20160094846A1 · Lee · 2016 [cited by applicant]
US 20160328643A1 · Liu et al. · 2016 [cited by applicant]
US 20160381386A1 · Krishnan et al. · 2016 [cited by applicant]
US 20170034538A1 · Lee · 2017 [cited by applicant]
US 20170249739A1 · Kallenberg et al. · 2017 [cited by applicant]
US 20170255832A1 · Jones et al. · 2017 [cited by applicant]
US 20190035113A1 · Salvi et al. · 2019 [cited by applicant]
US 20190138889A1 · Jiang et al. · 2019 [cited by applicant]
US 20190289321A1 · Liu et al. · 2019 [cited by applicant]
US 20190306526A1 · Cho et al. · 2019 [cited by applicant]
US 20190325273A1 · Kumar · 2019 [cited by applicant]
US 20200081431A1 · Weiss · 2020 [cited by applicant]
US 20210042558A1 · Choi et al. · 2021 [cited by applicant]
US 20210233328A1 · Krishnamurthy et al. · 2021 [cited by applicant]
US 20220284324A1 · Khalatian · 2022 [cited by examiner]
US 20230123820A1 · Wang et al. · 2023 [cited by applicant]
US 20230237628A1 · Gharbi et al. · 2023 [cited by applicant]
US 20230419507A1 · Planche et al. · 2023 [cited by applicant]
US 20250209568A1 · Dupeyron · 2025 [cited by examiner]
CN 116630366A · 2023 [cited by applicant]
WO 2016033458A1 · 2016 [cited by applicant]
Qiu et al., “CN 116112707 A Video processing method and device, electronic device and storage medium”. Date published May 12, 2023. [cited by examiner]
International Search Report and Written Opinion dated Nov. 7, 2024 for International Patent Application Number PCT/US2024/046919. [cited by applicant]
ISO/IEC 14496-10:2009, “Information technology—Coding of audio-visual objects—Part 10: Advanced Video Coding, Edition 5” May 13, 2009, Downloaded from the internet: http://www.iso.org/iso/iso_catalogue/catalogue_tc/cata… [cited by applicant]
Koutilya et al, “SharinGAN: Combining Synthetic and Real Data for Unsupervised Geometry Estimation”; Published in 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR); Date of Conference: Jun. 13-1… [cited by applicant]
Liao, J.Y.; Villasenor, J., “Adaptive intra block update for robust transmission of H.263” IEEE Transactions on Circuits and Systems for Video Technology, vol. 10, Issue 1, Date: Feb. 2000, pp. 30-35. [cited by applicant]
Nageswara Rao, G.; Gupta, P.S.S.K., “Improved Intra Prediction for Efficient Packetization in H.264 with Multiple Slice Groups”, 2007 IEEE International Conference on Multimedia and Expo, Date: Jul. 2-5, 2007, pp. 1607-… [cited by applicant]
Nunes, P.; Soares, L.D.; Pereira, F., “Error resilient macroblock rate control for H.264/AVC video coding” 15th IEEE International Conference on Image Processing, 2008. ICIP 2008. Date: Oct. 12-15, 2008, pp. 2132-2135. [cited by applicant]
Perronnin, Florent, “Output Embedding for Large Scale visual Recognition.” CVSPR tutorial: Large Scale Visual Recognition. Jun. 28, 2014. [cited by applicant]
Ranzato, Marc'Aurelio “Large Scale Visual Recognition Part IV: Deep Learning” Facebook A.I. research Jun. 28, 2014. [cited by applicant]
Schroff, Florian, et al. “FaceNet: A Unified Embedding for Face Recognition and Clustering.” [1503.03832] FaceNet: A Unified Embedding for Face Recognition and Clustering, Jun. 17, 2015. [cited by applicant]
Simonyan, Karen, and Andrew Zisserman. “Very Deep Convolutional Networks for Large-Scale Image Recognition.” [1409.1556] ICANN, Apr. 10, 2015. [cited by applicant]
Soomro et al., “UCF101: A Dataset of 101 Human Action Classes From Videos in the Wild”, Dec. 2012, arXiv.org, , p. 1-7. (Year: 2012). [cited by applicant]
Su et al., “Exploring Microscopic Fluctuation of Facial Expression for Mood Disorder Classification”, Dec. 2017, IEEE, 2017 International Conference of Orange Technologies (ICOT), p. 65-69. (Year: 2017). [cited by applicant]
Sun et al., “Learning a Convolutional Neural Network for Non-uniform Motion Blur Removal”, Jun. 2015, IEEE, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) p. 796-777. (Year: 2015). [cited by applicant]
Telecommunication Standardization Sector of ITU, International Telecommunication Union, Apr. 2013. [cited by applicant]
Vedaldi, Andrea “Input embedding, from shallow to deep” CVPR Tutorial on Large Scale Recognition 2014. [cited by applicant]
Vetro, A.; Peng Yin; Bede Liu; Huifang Sun, “Reduced spatio-temporal transcoding using an intra refresh technique” IEEE International Symposium on Circuits and Systems, 2002. ISCAS 2002. vol. 4, Date: 2002, pp. IV-723-I… [cited by applicant]
Wen-Nung Lie; Han-Ching Yeh; Zhi-Wei Gao; Ping-Chang Jui, “Error-Resilience Transcoding of H.264/AVC Compressed Videos” IEEE International Symposium on Circuits and Systems, 2007. ISCAS 2007. Date: May 27-30, 2007, pp. … [cited by applicant]
Worrall, S.T.; Sadka, A.H.; Sweeney, P.; Kondoz, A.M., “Motion adaptive error resilient encoding for MPEG-4” Proceedings. (ICASSP '01). 2001 IEEE International Conference on Acoustics, Speech, and Signal Processing, 200… [cited by applicant]
Yu-Kuang Tu*, Jar-Ferr Yang*, and Ming-Ting Sun, An Efficient Criterion for Mode Decision in H.264/AVC, 2006. [cited by applicant]
Zhang, R.; Regunathan, S.L.; Rose, K., “Video coding with optimal inter/intra-mode switching for packet loss resilience” IEEE Journal on Selected Areas in Communications, vol. 18, Issue 6, Date: Jun. 2000, pp. 966-976. [cited by applicant]
Zhenyu Wu; Boyce, J.M., “Optimal Frame Selection for H.264/AVC FMO Coding” 2006 IEEE International Conference on Image Processing, Date: Oct. 8-11, 2006, pp. 825-828. [cited by applicant]
Hochreiter et al., “Long Short-Term Memory,” Neural Computation, 1997, 9(8):1735-1780. [cited by applicant]
Chen et al. “Long-Term Video Interpolation with Bidirectional Predictive Network”, Dec. 2017, IEEE, 2017 IEEE Visual Communication and Image Processing (VCIP), p. 1-4 (Year: 2017). [cited by applicant]
Liu et al., “Video Frame Synthesis using Deep Voxel Flow”, Oct. 2017, IEEE, 2017 IEEE International Conference on Computer Vision (ICCV), p. 4473-4481. (Year: 2017). [cited by applicant]
Masci, Jonathan, et al. “Stacked Convolutional Auto-Encoders for Hierarchical Feature Extraction.” ICANN 2011, 2011, pp. 52-59., doi:10.1007/978-3-642-21735-7_7. [cited by applicant]
Mikolov, Tomas, et al. “Distributed Representations of Words and Phrases and Their Compositionality.” [1310.4546] Distributed Representations of Words and Phrases and Their Compositionality, Oct. 16, 2013. [cited by applicant]
Songyin Wu et al.: “ExtraSS: A Framework for Joint Spatial Super Sampling and Frame Extrapolation”, SA Conference Papers '23, Dec. 12-15, 2023, Sydney, NSW, Australia, © 2023, ACM ISBN 979-8-4007-0315-7/23/12. downloade… [cited by applicant]
Villegas et al., “Decomposing Motion and Content for Natural Video Sequence Prediction”, Jan. 8, 2018, arXiv.org, , p. 1-22. (Year:2018). [cited by applicant]
Wang, Zhangyang, et al. “Self-Tuned Deep Super Resolution.” [1504.05632] Self-Tuned Deep Super Resolution, Apr. 22, 2015,. [cited by applicant]
Xue et al. “Visual Dynamics: Probabilistic Future Frame Synthesis via Cross Convolutional Networks”, Dce 2016, ACM, NIPS'16: Proceedings of the 30th International Conference on Neural Information Processing Systems, p. … [cited by applicant]
Guo et al., “ExtraNet: Real-time Extrapolated Rendering for Low-latency Temporal Supersampling,” ACM Transactions on Graphics (TOG), Dec 10, 2021, 40(6):1-16. [cited by applicant]
Srivastava et al., “Unsupervised Learning of Video Representations using LSTMs,” CoRR, Submitted on Jan. 4, 2016, arXiv:1502-04681v3, 12 pages. [cited by applicant]