IP Library Granted Patent US 12,417,067
Granted Patent B2
US 12,417,067 · App. 17/963,722 · Granted Sep 16, 2025

Image processing

Inventors: Albert Saa-Garriga (Staines, GB); Daniel Valdez-Balderas (Staines, GB); Prashant Sharma (Staines, GB)
Assignee: SAMSUNG ELECTRONICS CO., LTD.
G06F3/147G06F3/1415G06T3/04G06T3/40G06T11/60G06V10/235G06V20/00G06T2210/22
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,417,067
App. No.
17/963,722
Granted
Sep 16, 2025
Kind
B2
Abstract

A method and system for processing an image is described. For example, the method comprises detecting a plurality of objects within an in input image; identifying dimensions of a display on which the input image is to be displayed; cropping the input image to obtain a cropped image which matches the identified dimensions, wherein the cropped image includes at least one of the plurality of detected objects; obtaining a list of missing objects which are not visible in the cropped image and which were detected in the input image; outputting a representation of each missing object in the list of missing objects to be displayed together with the cropped image; generating an updated image comprising the representation of at least one missing object and which matches the identified dimensions; and outputting the updated image to be displayed on the display.

Claims (50)

1. An image processing method comprising:

receiving an input image;

detecting a plurality of objects within the input image;

identifying dimensions of a display on which the input image is to be displayed;

cropping the input image to obtain a cropped image which matches the identified dimensions, wherein the cropped image is a portion of the input image and includes at least one of the plurality of detected objects;

obtaining a list of missing objects which are not visible in the cropped image and which were detected in the input image;

obtaining a representation of at least one missing object in the list of missing objects to be displayed together with the cropped image, wherein the representation of an object is at least one of a re-sized image of an object, a caption describing the object, or an icon;

generating an updated image comprising the representation of the at least one missing object and the cropped image; and

outputting the updated image to be displayed on the display.

2. The image processing method of claim 1 , wherein the updated image includes each representation at a location on the display which aligns with their location in the original input image.

3. The image processing method of claim 1 , further comprising:

receiving a selection signal of selecting at least one missing object.

4. The image processing method of claim 3 , wherein the updated image comprises the cropped image and the selected at least one missing object.

5. The image processing method of claim 3 , wherein the generating of the updated image comprises re-cropping the input image to obtain an updated cropped image comprising the selected at least one missing object.

6. The image processing method of claim 5 , wherein the updated cropped image is centered on one of the selected at least one missing object.

7. The image processing method of claim 1 , further comprising:

receiving the input image at a first device; and

identifying, using the first device, the dimensions of a display on a second device on which the input image is to be displayed.

8. The image processing method of claim 1 , wherein the cropped image is centered on one of the plurality of detected objects.

9. The image processing method of claim 8 , further comprising:

ranking the plurality of detected objects and

cropping the input image by centering on a highest ranked object.

10. The image processing method of claim 8 , further comprising:

training using historic selection signal of at least one missing object;

predicting, following training, a preference for an object in the plurality of detected objects; and

cropping the input image by centering on the predicted object with the highest preference.

11. The image processing method of claim 1 , wherein the identifying of the dimensions of the display comprises predicting the dimensions of the display based on historic preferences.

12. A non-transitory computer readable medium having computer readable program code which when running on a computer causes the computer to carry out the image processing method comprising:

receiving an input image;

detecting a plurality of objects within the input image;

identifying dimensions of a display on which the input image is to be displayed;

cropping the input image to obtain a cropped image which matches the identified dimensions, wherein the cropped image is a portion of the input image and includes at least one of the plurality of detected objects;

obtaining a list of missing objects which are not visible in the cropped image and which were detected in the input image;

obtaining a representation of at least one missing object in the list of missing objects to be displayed together with the cropped image, wherein the representation of an object is at least one of a re-sized image of an object, a caption describing the object, or an icon;

generating an updated image comprising the representation of the at least one missing object and the cropped image; and

outputting the updated image to be displayed on the display.

13. An electronic device comprising:

memory storing computer readable program code; and

a processor which executes the stored computer readable program code to carry out the image processing method comprising:

receiving an input image;

detecting a plurality of objects within the input image;

identifying dimensions of a display on which the input image is to be displayed;

cropping the input image to obtain a cropped image which matches the identified dimensions, wherein the cropped image is a portion of the input image and includes at least one of the plurality of detected objects;

obtaining a list of missing objects which are not visible in the cropped image and which were detected in the input image;

obtaining a representation of at least one missing object in the list of missing objects to be displayed together with the cropped image, wherein the representation of an object is at least one of a re-sized image of an object, a caption describing the object, or an icon;

generating an updated image comprising the representation of the at least one missing object and the cropped image; and

outputting the updated image to be displayed on the display.

14. A system comprising:

the electronic device of claim 13 , and

a second electronic device having a display on which the cropped image and representation of the at least one missing object are displayed.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Oct 11, 2022
From: SAA-GARRIGA, ALBERT; SHARMA, PRASHANT
To: SAMSUNG ELECTRONICS CO., LTD.
Reel/Frame 061381/0824 →
Priority Claims (2)
GB 2005277 · Apr 9, 2020 · national
GB 2013332 · Aug 26, 2020 · national
Continuity (2)
Continuation PCTKR2021004217 · Apr 5, 2021
Related Publication 20230036950A1 · Feb 2, 2023
References Cited (29)
US 8218895B1 · Gleicher et al. · 2012 [cited by applicant]
US 8861864B2 · Hong · 2014 [cited by examiner]
US 20060139371A1 · Lavine et al. · 2006 [cited by applicant]
US 20110096228A1 · Deigmoeller et al. · 2011 [cited by applicant]
US 20140270533A1 · Chedeau · 2014 [cited by applicant]
US 20140280180A1 · Edecker · 2014 [cited by examiner]
US 20160203585A1 · Welinder et al. · 2016 [cited by applicant]
US 20160269645A1 · Khoe et al. · 2016 [cited by applicant]
US 20170243356A1 · Eckert · 2017 [cited by applicant]
US 20180032238A1 · Jung · 2018 [cited by examiner]
US 20190370546A1 · Agarwal et al. · 2019 [cited by applicant]
US 20220245823A1 · Su · 2022 [cited by examiner]
Search Report (PCT/ISA/220, PCT/ISA/210) issued Jul. 28, 2021 by the International Searching Authority for International Patent Application No. PCT/KR2021/004217. [cited by applicant]
Written Opinion (PCT/ISA/237) issued Jul. 28, 2021 by the International Searching Authority for International Patent Application No. PCT/KR2021/004217. [cited by applicant]
Communication issued Feb. 19, 2021 by the Intellectual Property Office of Great Britain for British Patent Application No. GB 2013332.8. [cited by applicant]
He, et al., “Mask R-CNN” 2018, 12 pages total, arXiv:1703.06870v3 [cs.CV]. [cited by applicant]
Redmond, et al., “YOLOv3: An Incremental Improvement”, 2018, 6 pages total. [cited by applicant]
Boyla, et al., “YOLACT++ Better Real-time Instance Segmentation”, 2019, 12 pages total, arXiv:1912.06218v1 [cs.CV]. [cited by applicant]
Lu, et al., “An End-to-End Neural Network for Image Cropping by Learning Composition from Aesthetic Photos”, 2019, 13 pages total, arXiv:1907.01432v3 [cs.CV]. [cited by applicant]
Li, et al., “A2-RL: Aesthetics Aware Reinforcement Learning for Image Cropping”, 2018, 9 pages total. [cited by applicant]
Zeng et al., “Reliable and Efficient Image Cropping: A Grid Anchor based Approach”, 2019, 9 pages total, arXiv:1904.04441v1 [cs.CV]. [cited by applicant]
Cornia, et al., “Meshed-Memory Transformer for Image Captioning”, 2020, 15 pages total, arXiv:1912.08226v2 [cs.CV]. [cited by applicant]
Yang, et al., “Image Captioning with Object Detection and Localization”, 2017, 10 pages total. [cited by applicant]
Tan, et al., “Cycle-IR: Deep Cyclic Image Retargeting”, 2019, 12 pages total, arXiv:1905.03556v1 [cs.CV]. [cited by applicant]
Lin et al., “DeepIR: A Deep Semantics Driven Framework for Image Retargeting”, 2019, 9 pages total, arXiv:1811.07793v3 [cs.CV]. [cited by applicant]
Tang, et al., “Image Retargetability”, 2019, IEEE Transactions on Multimedia, vol. XX, Issue XX, 13 pages total, arXiv:1802.04392v2 [cs.CV]. [cited by applicant]
Kim et al., “Deep Video Inpainting”, 2019, 10 pages total, arXiv:1905.01639v1 [cs.CV]. [cited by applicant]
“Distribution of Instagram users worldwide as of Apr. 2022, by age group” 2022, 1 page total, statista.com https://www.statista.com/statistics/325587/instagram-global-age-group/. [cited by applicant]
Chen, et al., “Automatic Image Cropping : A Computational Complexity Study”, 2016, 9 pages total. [cited by applicant]