IP Library Granted Patent US 12,412,253
Granted Patent B2
US 12,412,253 · App. 17/567,927 · Granted Sep 9, 2025

Contextual priority based multimedia modification

Inventors: Sathya Santhar (Ramapuram, IN); Sarbajit K. Rakshit (Kolkata, IN); Sridevi Kannan (Chennai, IN); Samuel Mathew Jawaharlal (Chennai, IN)
Assignee: International Business Machines Corporation
G06T5/77G06N3/045G06V10/764G06V20/20G06T2207/10016G06T2207/20084
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,412,253
App. No.
17/567,927
Granted
Sep 9, 2025
Kind
B2
Abstract

A computer-implemented method for multimedia modification is disclosed. The computer-implemented method includes classifying one or more objects detected within a user's field of view through an augmented reality environment. The computer-implemented method further includes determining a context of the user based, at least in part, on the one or more classified objects detected within the user's field of view. The computer-implemented method further includes generating a priority score for the one or more classified objects based, at least in part, on the context of the user. The computer-implemented method further includes modifying an object detected within the user's field of view based, at least in part, on the priority score of the object.

Claims (68)

1. A computer-implemented method for multimedia modification, the computer-implemented method comprising:

classifying one or more objects detected within a user's field of view through an augmented reality environment;

determining a context of the user based, at least in part, on the one or more classified objects detected within the user's field of view;

generating a priority score for the one or more classified objects based, at least in part, on the context of the user and a precaution taken by the user in the context;

modifying an object detected within the user's field of view based, at least in part, on the priority score of the object; and

contextually prioritizing the one or more classified objects based, at least in part, on the priority score to regenerate an image showing a high priority object.

2. The computer-implemented method of claim 1 , wherein modifying an object detected within the user's field of view includes:

determining that a priority score of the object is below a predetermined threshold; and

removing the object from the user's field of view within the augmented reality environment.

3. The computer-implemented method of claim 2 , wherein removing the object from the user's field of view further includes:

generating, using a generative adversarial network, a portion of a background of the user's field of view corresponding to an area encompassed by the object; and

displaying the generated portion of the background corresponding to the area encompassed by the object within the augmented reality environment.

4. The computer-implemented method of claim 1 , wherein modifying an object detected within the user's field of view includes:

determining that a priority score of the object is above a predetermined threshold;

determining that a portion of the object is blocked from the user's view by another object having a priority score below the predetermined threshold;

generating, using a generative adversarial network, the portion of the object blocked from the user's view; and

displaying the generated portion of the object blocked from the user's view within the augmented reality environment.

5. The computer-implemented method of claim 1 , wherein modifying an object detected within the user's field of view is further based, at least in part, on:

comparing a classification of the object to object modification policies associated with the user; and

selecting one of the object modification policies based on the context of the user and the priority score of the object.

6. The computer-implemented method of claim 1 , wherein generating a priority score for the one or more classified objects is further based, at least in part, on one or more of: a distance between a classified object and the user, a degree of visibility of an object, a detected motion of an object relative to the user, a perceived danger associated with an object, or one or more detected precautions taken by the user with respect to a classified object.

7. The computer-implemented method of claim 1 , wherein an object is modified within the augmented reality environment using a generative adversarial network.

8. A computer program product for multimedia modification, the computer program product comprising one or more computer readable storage media and program instructions stored on the one or more computer readable storage media, the program instructions including instructions to:

classify one or more objects detected within a user's field of view through an augmented reality environment;

determine a context of the user based, at least in part, on the one or more classified objects detected within the user's field of view;

generate a priority score for the one or more classified objects based, at least in part, on the context of the user and a precaution taken by the user in the context;

modify an object detected within the user's field of view based, at least in part, on the priority score of the object; and

contextually prioritize the one or more classified objects based, at least in part, on the priority score to regenerate an image showing a high priority object.

9. The computer program product of claim 8 , wherein the instructions to modify an object detected within the user's field of view includes instructions to:

determine that a priority score of the object is below a predetermined threshold; and

remove the object from the user's field of view within the augmented reality environment.

10. The computer program product of claim 9 , wherein the instructions to remove the object from the user's field of view further includes instructions to:

generate, using a generative adversarial network, a portion of a background of the user's field of view corresponding to an area encompassed by the object; and

display the generated portion of the background corresponding to the area encompassed by the object within the augmented reality environment.

11. The computer program product of claim 8 , wherein the instructions to modify an object detected within the user's field of view includes instructions to:

determine that a priority score of the object is above a predetermined threshold;

determine that a portion of the object is blocked from the user's view by another object having a priority score below the predetermined threshold;

generate, using a generative adversarial network, the portion of the object blocked from the user's view; and

display the generated portion of the object blocked from the user's view within the augmented reality environment.

12. The computer program product of claim 8 , wherein the instructions to modify an object detected within the user's field of view is further based, at least in part, on instructions to:

compare a classification of the object to one or more object modification policies associated with the user.

13. The computer program product of claim 8 , wherein the instructions to generate a priority score for the one or more classified objects is further based, at least in part, on one or more of: a degree of visibility of an object, a detected motion of an object relative to the user, a perceived danger associated with an object, or one or more detected precautions taken by the user with respect to a classified object.

14. The computer program product of claim 8 , wherein an object is modified within the augmented reality environment using a generative adversarial network.

15. A computer system for multimedia modification, comprising:

one or more computer processors;

one or more computer readable storage media;

computer program instructions;

the computer program instructions being stored on the one or more computer readable storage media for execution by the one or more computer processors; and

the computer program instructions including instructions to:

classify one or more objects detected within a user's field of view through an augmented reality environment;

determine a context of the user based, at least in part, on the one or more classified objects detected within the user's field of view;

generate a priority score for the one or more classified objects based, at least in part, on the context of the user and a precaution taken by the user in the context;

modify an object detected within the user's field of view based, at least in part, on the priority score of the object; and

contextually prioritize the one or more classified objects based, at least in part, on the priority score to regenerate an image showing a high priority object.

16. The computer system of claim 15 , wherein the instructions to modify an object detected within the user's field of view includes instructions to:

determine that a priority score of the object is below a predetermined threshold; and

remove the object from the user's field of view within the augmented reality environment.

17. The computer system of claim 16 , wherein the instructions to remove the object from the user's field of view further includes instructions to:

generate, using a generative adversarial network, a portion of a background of the user's field of view corresponding to an area encompassed by the object; and

display the generated portion of the background corresponding to the area encompassed by the object within the augmented reality environment.

18. The computer system of claim 15 , wherein the instructions to modify an object detected within the user's field of view includes instructions to:

determine that a priority score of the object is above a predetermined threshold;

determine that a portion of the object is blocked from the user's view by another object having a priority score below the predetermined threshold;

generate, using a generative adversarial network, the portion of the object blocked from the user's view; and

display the generated portion of the object blocked from the user's view within the augmented reality environment.

19. The computer system of claim 15 , wherein the instructions to modify an object detected within the user's field of view is further based, at least in part, on instructions to:

compare a classification of the object to one or more object modification policies associated with the user.

20. The computer system of claim 15 , wherein the instructions to generate a priority score for the one or more classified objects is further based, at least in part, on one or more of: a distance between a classified object and the user, a degree of visibility of an object, a detected motion of an object relative to the user, a perceived danger associated with an object, or one or more detected precautions taken by the user with respect to a classified object.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jan 4, 2022
From: SANTHAR, SATHYA; RAKSHIT, SARBAJIT K.; KANNAN, SRIDEVI; JAWAHARLAL, SAMUEL MATHEW
To: INTERNATIONAL BUSINESS MACHINES CORPORATION
Reel/Frame 058535/0619 →
Continuity (1)
Related Publication 20230214966A1 · Jul 6, 2023
References Cited (14)
US 11551444B2 · Gibbon · 2023 [cited by examiner]
US 20180089506A1 · Taite · 2018 [cited by applicant]
US 20180160100A1 · Konttori · 2018 [cited by applicant]
US 20190236759A1 · Lai · 2019 [cited by examiner]
US 20190251401A1 · Shechtman · 2019 [cited by examiner]
US 20210377548A1 · Lakshmanan · 2021 [cited by examiner]
WO 2012082444A1 · 2012 [cited by applicant]
WO WO2023022373A1 · 2023 [cited by examiner]
“Profile Development to Predict Content Adaptations”, An IP.com Prior Art Database Technical Disclosure, Authors et. al.: Disclosed Anonymously, IP.com No. IPCOM000265736D, IP.com Electronic Publication Date: May 11, 20… [cited by applicant]
Lin et al., “ST-GAN: Spatial Transformer Generative Adversarial Networks for Image Compositing”, arXiv:1803.01837v1 [cs.CV] Mar. 5, 2018, 13 pages. [cited by applicant]
Mell et al., “The NIST Definition of Cloud Computing”, Recommendations of the National Institute of Standards and Technology, NIST Special Publication 800-145, Sep. 2011, 7 pages. [cited by applicant]
Redmon et al., “You Only Look Once: Unified, Real-Time Object Detection”, arXiv:1506.02640v5 [cs.CV] May 9, 2016, 10 pages. [cited by applicant]
Sylvain et al., “Object-Centric Image Generation from Layouts”, arXiv:2003.07449v2 [cs.CV] Dec. 3, 2020, 13 pages. [cited by applicant]
Zhu et al., “Moving Object Detection Based on Background Compensation and Deep Learning”, Received: Nov. 3, 2020; Accepted: Nov. 23, 2020; Published: Nov. 27, 2020, Symmetry 2020, 12, 1965; doi:10.3390/sym12121965, 17 p… [cited by applicant]