IP Library › Granted Patent US 12,375,678
Granted Patent B2
US 12,375,678 · App. 18/477,920 · Granted Jul 29, 2025

Methods and non-transitory computer readable storage medium for pre-analysis based resampling compression for machine vision

Inventors: Shurun Wang (Beijing, CN); Zhao Wang (Beijing, CN); Yan Ye (San Diego, CA); Shiqi Wang (Hong Kong, CN)
Assignee: Alibaba Damo (Hangzhou) Technology Co., Ltd.
H04N19/132H04N19/119H04N19/124H04N19/136H04N19/23H04N19/463H04N19/85
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,375,678
App. No.
18/477,920
Granted
Jul 29, 2025
Kind
B2
Abstract

An image data encoding method is provided. The image data encoding method includes receiving an input image and a quantization parameter (QP); performing pre-analysis of the input image based on the QP to obtain input image data for machine version; determining at least one of a temporal redundancy or a spatial redundancy of the input image data; determining, based on the at least one of temporal redundancy or spatial redundancy, whether to perform at least one of temporal down-sampling or spatial down-sampling; performing down-sampling of the input image data according to the determination, wherein the down-sampling includes at least one of temporal down-sampling or spatial down-sampling; and encoding the down-sampled image data based on the QP.

Claims (77)

1. An image data encoding method, comprising:

receiving an input image and a quantization parameter (QP);

performing pre-analysis of the input image based on the QP to obtain a pre-analyzed input image for machine vision;

determining at least one of a temporal redundancy or a spatial redundancy of the pre-analyzed input image;

determining, based on the at least one of temporal redundancy or spatial redundancy, whether to perform at least one of temporal down-sampling or spatial down-sampling;

performing down-sampling on the pre-analyzed input image according to the determination, wherein the down-sampling includes at least one of temporal down-sampling or spatial down-sampling, and performing down-sampling on the pre-analyzed input image according to the determination further comprises:

performing temporal down-sampling based on the determination of temporal redundancy; or

performing spatial down-sampling based on the determination of spatial redundancy; and

encoding the down-sampled image based on the QP.

2. The method according to claim 1 , wherein determining at least one of a temporal redundancy or a spatial redundancy of the pre-analyzed input image comprises:

determining a temporal complexity of the pre-analyzed input image, the temporal complexity indicating the temporal redundancy of the pre-analyzed input image; and

wherein performing temporal down-sampling based on the determination of temporal redundancy comprises:

in response to the temporal complexity being smaller than a threshold value, performing the temporal down-sampling.

3. The method according to claim 1 , wherein performing pre-analysis of the input image based on the QP to obtain the pre-analyzed input image for machine vision further comprises:

performing instance segmentation of the input image, to determine an instance area and a background of the input image.

4. The method according to claim 3 , wherein determining at least one of a temporal redundancy or a spatial redundancy of the pre-analyzed input image comprises:

determining a difference between the instance area and the background, the difference indicating the spatial redundancy of the pre-analyzed input image; and

performing spatial down-sampling based on the determination of spatial redundancy comprises:

in response to the difference being greater than a threshold value, performing the spatial down-sampling.

5. The method according to claim 4 , wherein determining the difference between the instance area and the background, the difference indicating the spatial redundancy of the pre-analyzed input image further comprises:

determining a portion parameter; and

in response to the difference being greater than a threshold value, performing the spatial down-sampling, further comprises:

in response to the portion parameter being greater than a threshold value, performing the spatial down-sampling; wherein the portion parameter is obtained by dividing an area of the input image by an area of objects.

6. The method according to claim 1 , further comprising:

in response to performing the temporal down-sampling, signaling a first flag indicating temporal down-sampling is performed.

7. The method according to claim 1 , further comprising:

in response to performing the spatial down-sampling, signaling a second flag indicating spatial down-sampling is performed and an index indicating a resampling factor.

8. The method according to claim 1 , further comprising:

generating resampling control information associated with the at least one of temporal down-sampling or spatial down-sampling; and

encoding the resampling control information in a bitstream.

9. An image data decoding method, comprising:

receiving a bitstream comprising image data, wherein the image data is obtained by encoding a down-sampled image based on a quantization parameter (QP), the down-sampled image is obtained by performing down-sampling on a pre-analyzed input image, and the pre-analyzed input image is obtained by performing pre-analysis of the input image based on the QP;

generating reconstructed image;

determining whether to perform at least one of temporal up-sampling or spatial up-sampling of the reconstructed image based on the bitstream; and

performing, based on the determination, the at least one of temporal up-sampling or spatial up-sampling to obtain a recovered image.

10. The method according to claim 9 , wherein determining whether to perform at least one of temporal up-sampling or spatial up-sampling on the reconstructed image based on the bitstream further comprises:

performing the temporal up-sampling when a first flag indicating temporal down-sampling has been performed exists in the bitstream; and

performing the spatial up-sampling when a second flag indicating spatial down-sampling has been performed exists in the bitstream.

11. The method according to claim 10 , wherein performing the spatial up-sampling when a second flag indicating spatial down-sampling has been performed exists in the bitstream further comprises:

decoding an index indicating a resampling factor; and

performing the spatial up-sampling based on the resampling factor.

12. The method according to claim 9 , further comprising:

decoding resampling control information associated with the at least one of temporal up-sampling or spatial up-sampling; and

performing the at least one of temporal up-sampling or spatial up-sampling based on the resampling control information.

13. A non-transitory computer readable storage medium storing a set of instructions that are executable by one or more processors of a system to cause the system to perform operations for generating a bitstream, the operations comprising:

receiving a video sequence;

encoding one or more pictures of the video sequence by:

receiving an input image and a quantization parameter (QP);

performing pre-analysis on the input image based on the QP to obtain a pre-analyzed input image for machine vision;

determining at least one of a temporal redundancy or a spatial redundancy of the pre-analyzed input image;

determining, based on the at least one of temporal redundancy or spatial redundancy, whether to perform at least one of temporal down-sampling or spatial down-sampling;

performing down-sampling on the pre-analyzed input image according to the determination, wherein the down-sampling includes at least one of temporal down-sampling or spatial down-sampling, and performing down-sampling on the pre-analyzed input image according to the determination further comprises:

performing temporal down-sampling based on the determination of temporal redundancy; or

performing spatial down-sampling based on the determination of spatial redundancy; and

encoding the down-sampled image based on the QP; and

generating a bitstream that is based on the encoding.

14. The non-transitory computer readable storage medium according to claim 13 , wherein determining at least one of a temporal redundancy or a spatial redundancy of the pre-analyzed input image comprises:

determining a temporal complexity of the pre-analyzed input image, the temporal complexity indicating the temporal redundancy of the pre-analyzed input image; and

performing temporal down-sampling based on the determination of temporal redundancy comprises:

in response to the temporal complexity being smaller than a threshold value, performing the temporal down-sampling.

15. The non-transitory computer readable storage medium according to claim 13 , wherein performing pre-analysis on the input image based on the QP to obtain pre-analyzed input image for machine vision further comprises:

performing instance segmentation of the input image, to determine an instance area and a background of the input image.

16. The non-transitory computer readable storage medium according to claim 15 , wherein determining at least one of a temporal redundancy or a spatial redundancy of the pre-analyzed input image comprises:

determining a difference between the instance area and the background, the difference indicating the spatial redundancy of the pre-analyzed input image; and

performing spatial down-sampling based on the determination of spatial redundancy comprises:

in response to the difference being greater than a threshold value, performing the spatial down-sampling.

17. The non-transitory computer readable storage medium according to claim 16 , wherein determining the difference between the instance area and the background, the difference indicating the spatial redundancy of the pre-analyzed input image further comprises:

determining a portion parameter; and

in response to the difference being greater than a threshold value, performing the spatial down-sampling, further comprises:

in response to the portion parameter being greater than a threshold value, performing the spatial down-sampling; wherein the portion parameter is obtained by dividing an area of the input image by an area of objects.

18. The non-transitory computer readable storage medium according to claim 13 , wherein the operations further comprise:

in response to performing the temporal down-sampling, signaling a first flag indicating performance of temporal down-sampling.

19. The non-transitory computer readable storage medium according to claim 13 , wherein the operations further comprise:

in response to performing the spatial down-sampling, signaling a second flag indicating performance of spatial down-sampling and an index indicating a resampling factor.

20. The non-transitory computer readable storage medium according to claim 13 , wherein the operations further comprise:

generating resampling control information associated with the at least one of temporal down-sampling or spatial down-sampling; and

encoding the resampling control information in the bitstream.

Assignments (3)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Aug 5, 2026
From: ALIBABA DAMO (HANGZHOU) TECHNOLOGY CO., LTD.
To: ALIBABA INNOVATION PRIVATE LIMITED
Reel/Frame 075529/0583 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Aug 5, 2026
From: ALIBABA INNOVATION PRIVATE LIMITED
To: SIM IP 5 LLC
Reel/Frame 075529/0713 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jan 24, 2024
From: WANG, SHURUN; WANG, ZHAO; YE, YAN; WANG, SHIQI
To: ALIBABA DAMO (HANGZHOU) TECHNOLOGY CO., LTD.
Reel/Frame 066233/0736 →
Continuity (2)
Provisional Application 63378891 · Oct 10, 2022
Related Publication 20240121395A1 · Apr 11, 2024
References Cited (59)
US 6591017B1 · Fukuhara · 2003 [cited by examiner]
US 20090198827A1 · Hughes · 2009 [cited by examiner]
US 20130322524A1 · Jang et al. · 2013 [cited by applicant]
US 20140010294A1 · Ye · 2014 [cited by examiner]
US 20150098510A1 · Ye · 2015 [cited by examiner]
US 20150103886A1 · He · 2015 [cited by examiner]
US 20150172616A1 · Ye · 2015 [cited by examiner]
US 20150326857A1 · Zhang et al. · 2015 [cited by applicant]
US 20160330457A1 · Ye · 2016 [cited by examiner]
US 20200162751A1 · Kim et al. · 2020 [cited by applicant]
US 20210067803A1 · Li et al. · 2021 [cited by applicant]
US 20220094962A1 · Choi · 2022 [cited by examiner]
US 20220210402A1 · Li · 2022 [cited by examiner]
US 20220224900A1 · Li · 2022 [cited by examiner]
US 20220224924A1 · Li · 2022 [cited by examiner]
CN 101312529 · 2007 [cited by examiner]
CN 113379753 · 2020 [cited by examiner]
CN 113313776 · 2021 [cited by examiner]
CN 113852816 · 2021 [cited by examiner]
CN 114342371A · 2022 [cited by applicant]
JP 2011004343A · 2011 [cited by applicant]
WO 2021228513A1 · 2021 [cited by applicant]
WO WO2023240835 · 2022 [cited by examiner]
Li, Song-nan translation CN 113379753 Mar. 10, 2020 (Year: 2020). [cited by examiner]
Deng, Xuan translation CN 113313776 May 27, 2021 (Year: 2021). [cited by examiner]
Liu, Yu translation CN 113852816 Sep. 28, 2021 (Year: 2021). [cited by examiner]
Wang, Ronggang translation WO 2023240835 Sep. 26, 2022 (Year: 2022). [cited by examiner]
CN 101312529 translation Peng, Li-zhong May 24, 2007 (Year: 2007). [cited by examiner]
Adini et al., “Context-enabled learning in the human visual system,” Nature, vol. 415, 2002, pp. 790-793. [cited by applicant]
Afonso et al., “Video Compression Based on Spatio-Temporal Resolution Adaptation,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 29, No. 1, 2019 pp. 275-280. [cited by applicant]
An et al., “Block partitioning structure for next generation video coding,” International Telecommunications Union, 2015, 8 pages. [cited by applicant]
Balle, et al., “Variational Image Compression with a Scale Hyperprior,” International Conference on Learning Representations, 2018, 23 pages. [cited by applicant]
Balle et al., End-to-End Optimized Image Compression, ICLR, 2017, 27 pages. [cited by applicant]
Bosch et al., “Segmentation-Based Video Compression Using Texture and Motion Models,” IEEE Journal of Selected Topics in Signal Processing, vol. 5, No. 7, 2011, pp. 1366-1377. [cited by applicant]
Bross et al., Developments in International Video Coding Standardization After AVC, With an Overview of Versatile Video Coding (VVC), Proceedings of the IEEE, vol. 109, No. 9, Sep. 2021, pp. 1463-1493. [cited by applicant]
Brouard, Olivier, “Pre-analysis of video for its advanced coding. Application to the HDTV coding in H.264 streams,” Université de Nantes, 48 pages, 2010. [cited by applicant]
Chao, et al., “A Novel Rate Control Framework for SIFT/SURF Feature Preservation in H.264/AVC Video Compression,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 25, No. 6, Jun. 2015, pp. 958-972. [cited by applicant]
Duan et al., “Overview of the MPEG-CDVS Standard,” IEEE Transactions on Image Processing, vol. 25, No. 1, Jan. 2016, pp. 179-194. [cited by applicant]
Duan et al., Compact Descriptors for Video Analysis: The Emerging MPEG Standard, IEEE Computer Society, 2018, pp. 44-54. [cited by applicant]
Duan et al., “Video Coding for Machines: A Paradigm of Collaborative Compression and Intelligent Analytics,” IEEE Transactions on Image Processing, vol. 29, 2020, pp. 8680-8695. [cited by applicant]
Garcia-Lucas et al., “Acceleration of the integer motion estimation in JEM through pre-analysis techniques,” J. Supercomput, 75:1203-1214, 2019. [cited by applicant]
Jiang et al., “An End-to-End Compression Framework Based on Convolutional Neural Networks,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 28, No. 10, Oct. 2018, pp. 3007-3018. [cited by applicant]
Li et al., “A Domain Rate Control Algorithm for High Efficiency Video Coding,” IEEE Transactions on Image Processing, vol. 23, No. 9, Sep. 2014, pp. 3841-3854. [cited by applicant]
Lin et al., “Adaptive Downsampling to Improve Image Compression at Low Bit Rates,” IEEE Transactions on Image Processing, vol. 15, No. 9, Sep. 2006, pp. 2513-2521. [cited by applicant]
Liu et al., “Compressive Sampling-Based Image Coding for Resource-Deficient Visual Communication,” IEEE Transactions on Image Processing, vol. 25, No. 6, Jun. 2016, pp. 2844-2855. [cited by applicant]
Liu et al., “CNN-Basesd DCT-Like Transform for Image Compression,” Springer, 2018, pp. 61-72. [cited by applicant]
Minnen et al., “Joint Autoregressive and Hierarchical Priors for Learned Image Compression,” 32nd Conference on Neural Information Processing System, 2018, 10 pages. [cited by applicant]
Mohan et al., “Internet of Video Things in 2030: A World with Many Cameras,” IEEE Xplore, 2017, 4 pages. [cited by applicant]
Pfaff et al., “CE3: Affine linear weighted intra prediction (CE3-4.1, CE3-4.2),” JVET-N0217, 14 [cited by applicant]
Rabbani et al., “JPEG2000: Image compression fundamentals, standards and practice,” Journal of Electronic Imaging, 2002, 11(2): 286. [cited by applicant]
Sullivan et al., “Overview of the High Efficiency Video Coding (HEVC) Standard,” IEE Transactions on Circuits and Systems for Video Technology, vol. 22, No. 12, pp. 1649-1668 (2012). [cited by applicant]
Toderici et al., “Variable Rate Image Compression with Recurrent Neural Networks,” ICLR, 2016, 12 pages. [cited by applicant]
Wallace et al., “The JPEG Still Picture Compression Standard,” IEEE Transactions on Consumer Electronics, vol. 38, No. 1, Feb. 1992, 17 pages. [cited by applicant]
Wang et al., “Extended Coding Unit Partitioning for Future Video Coding,” IEEE Transactions on Image Processing, vol. 29, 2020, pp. 2931-2946. [cited by applicant]
Wiegand et al., “Overview of the H.264/AVC Video Coding Standard,” IEEE Transactions on Circuitds and Systems for Video Technology, vol. 13, No. 7, Jul. 2003, pp. 560-576. [cited by applicant]
Yokoyama et al., “A Rate Control Method With Pre-Analysis For Real-Time MPEG-2 Video Coding,” 2001 International Conference on Image Processing, IEEE, pp. 514-517, 2001. [cited by applicant]
Yuxin et al., https://github.com/facebookresearch/detectron2. Detectron2, 2019. [cited by applicant]
Zhao et al., “Mode-dependent non-separable secondary transform,” ITU-T SG16/Q6 Doc. COM16-C1044, 5 pages, 2015. [cited by applicant]
PCT International Search Report and Written Opinion mailed Jan. 18, 2024, issued in corresponding International Application No. PCT/CN2023/123843 (8 pgs.). [cited by applicant]