IP Library Granted Patent US 11,381,715
Granted Patent B2
US 11,381,715 · App. 16/450,751 · Granted Jul 5, 2022

Computer method and apparatus making screens safe for those with photosensitivity

Inventors: Andrei Barbu (Cambridge, MA); Dalitso Banda (Cambridge, MA); Boris Katz (Cambridge, MA)
Assignee: MASSACHUSETTS INSTITUTE OF TECHNOLOGY
H04N5/21A61B5/163A61B5/369A61B5/7267G06T5/001G06T7/0002A61B2503/12G06T2207/10004G06T2207/10016G06T2207/20081G06T2207/20084G06T2207/30168
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,381,715
App. No.
16/450,751
Granted
Jul 5, 2022
Kind
B2
Abstract

In an embodiment, a method, and corresponding system and non-transitory computer readable medium storing instructions configured to cause a processor to execute steps are configured to introduce an auxiliary transformation to a digital media, resulting in a transformed digital media by generating the auxiliary transformation with a transform function. The method is further configured to evaluate the transformed digital media to generate a metric estimating a human response to the transformed digital media altered by the introduced auxiliary transformation. The method is further configured to train a neural network to remove the auxiliary transformation from any digital media by learning a desired transformation function from the transformed digital media and the metric associated with the transformed digital media.

Claims (42)

1. A method comprising:

introducing an auxiliary transformation to a digital media by applying a transform function to the digital media, resulting in a transformed digital media that represents the digital media altered by the transform function;

evaluating the transformed digital media to generate a metric estimating a human or animal response to the transformed digital media altered by the introduced auxiliary transformation; and

training a machine learning model by learning a desired transformation function from the transformed digital media and the metric associated with the transformed digital media, the resulting machine learning model able to remove the auxiliary transformation from any digital media by applying the learned desired transformation function.

2. The method of claim 1 , wherein the desired transformation function is an inverse function of the auxiliary transformation.

3. The method of claim 1 , wherein introducing the auxiliary transformation further includes introducing a respective auxiliary transformation to each digital media of a collection of original digital media, resulting in a collection of transformed digital media by generating the auxiliary transformation with the auxiliary transform function.

4. The method of claim 1 , wherein the auxiliary transform function is specified by a designer.

5. The method of claim 1 , wherein the auxiliary transform function is learned by a second machine learning model.

6. The method of claim 1 , wherein generating the metric estimating the human response to the transformed digital media includes employing a predefined function to estimate the human response based on the transformed media.

7. The method of claim 1 , wherein generating the metric estimating the human response to the transformed digital media includes collecting at least one physiological measurement, behavioral measurement, or preference from at least one testing user experiencing the transformed digital media.

8. The method of claim 1 , wherein generating the metric estimating the human response to the transformed digital media includes employing a second machine learning model that is trained as a proxy for a human physiological measurement or behavioral measurement.

9. The method of claim 1 , wherein the media type of the collection of digital media is at least one of videos, images, audio, text, virtual reality scenes, augmented reality scenes, three-dimensional (3D) video, and 3D scenes.

10. The method of claim 1 , wherein the auxiliary transformation is at least one of:

introduction of flashing, a modification of features representing emotions in an image or video, a highlighting of features representing emotions in an image or video, or an obscuring of features that are irrelevant to emotions in an image or video, highlighting or obscuring features in the media representing a distraction, highlighting a key behavior out of a sequence of actions, highlighting future behaviors for the user or other agents, highlighting features indicating social interactions, a modification of text, human annotations, and modification of audio features.

11. The method of claim 1 , further comprising:

repairing a given media to a modified media by applying the desired function to the given media, wherein the modified media is generated by application of the desired function to cause a human response or physiological state estimated by a metric having a particular value.

12. The method of claim 11 , further comprising:

adjusting the machine learning model based on further behavioral measurements, physiological measurements, or preferences from a user experiencing the modified media.

13. The method of claim 1 , wherein the machine learning model is a neural network.

14. A system comprising:

a transformation module configured to an auxiliary transformation to a digital media by applying a transform function to the digital media, resulting in a transformed digital media that represents the digital media altered by the transform function;

an evaluation module configured to evaluate the transformed digital media to generate a metric estimating a human or animal response to the transformed digital media altered by the introduced auxiliary transformation; and

an inverse function generation module configured to train a machine learning model by learning a desired transformation function from the transformed digital media and the metric associated with the transformed digital media, the resulting machine learning model able to remove the auxiliary transformation from any digital media by applying the learned desired transformation function.

15. The system of claim 14 , wherein the desired transformation function is an inverse function of the auxiliary transformation.

16. The system of claim 14 , wherein introducing the auxiliary transformation further includes introducing a respective auxiliary transformation to each digital media of a collection of original digital media, resulting in a collection of transformed digital media by generating the auxiliary transformation with the auxiliary transform function.

17. The system of claim 14 , wherein the auxiliary transform function is specified by a designer.

18. The system of claim 14 , wherein the auxiliary transform function is learned by a second machine learning model.

19. The system of claim 14 , wherein the evaluation metric is further configured to generate the metric estimating the human response to the transformed digital media by employing a predefined function to estimate the human response based on the transformed media.

20. The system of claim 14 , wherein the evaluation metric is further configured to generate the metric estimating the human response to the transformed digital media includes collecting at least one physiological measurement, behavioral measurement, or preference from at least one testing user experiencing the transformed digital media.

21. The system of claim 14 , wherein the transformation module is further configured to generate the metric estimating the human response to the transformed digital media by employing a second machine learning model that is trained as a proxy for a human physiological measurement or behavioral measurement.

22. The system of claim 14 , wherein the media type of the collection of digital media is at least one of videos, images, audio, text, virtual reality scenes, augmented reality scenes, three-dimensional (3D) video, and 3D scenes.

23. The system of claim 14 , wherein the auxiliary transformation is at least one of:

introduction of flashing, a modification of features representing emotions in an image or video, a highlighting of features representing emotions in an image or video, or an obscuring of features that are irrelevant to emotions in an image or video, highlighting or obscuring features in the media representing a distraction, highlighting a key out of a sequence of actions, highlighting future behaviors for the user or other agents, highlighting features indicating social interactions, a modification of text, human annotations, and modification of audio features.

24. The system of claim 14 , further comprising:

a repair module configured to repair a given media to a modified media by applying the desired function to the given media, wherein the modified media is generated by application of the desired function to cause a human response estimated by a metric having a particular value.

25. The system of claim 24 , further comprising:

adjusting the machine learning model based on further behavioral measurements, physiological measurements, or preferences from a user experiencing the modified media.

26. The system of claim 14 , wherein the machine learning model is a neural network.

27. A non-transitory computer-readable medium configured to store instructions for training a machine learning model, the instructions, when loaded and executed by a processor, causes the processor to:

introduce an auxiliary transformation to a digital media by applying a transform function to the digital media, resulting in a transformed digital media that represents the digital media altered by the transform function;

evaluate the transformed digital media to generate a metric estimating a human or animal response to the transformed digital media altered by the introduced auxiliary transformation; and

train a machine learning model by learning a desired transformation function from the transformed digital media and the metric associated with the transformed digital media, the resulting machine learning model able to remove the auxiliary transformation from any digital media by applying the learned desired transformation function.

Assignments (3)
CONFIRMATORY LICENSE Recorded Dec 14, 2022
From: MASSACHUSETTS INSTITUE OF TECHNOLOGY
To: NATIONAL SCIENCE FOUNDATION
Reel/Frame 062122/0741 →
CONFIRMATORY LICENSE Recorded Nov 2, 2020
From: MASSACHUSETTS INSTITUTE OF TECHNOLOGY
To: NATIONAL SCIENCE FOUNDATION
Reel/Frame 054280/0280 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jul 11, 2019
From: BARBU, ANDREI; BANDA, DALITSO; KATZ, BORIS
To: MASSACHUSETTS INSTITUTE OF TECHNOLOGY
Reel/Frame 049724/0011 →
Continuity (2)
Provisional Application 62698652 · Jul 16, 2018
Related Publication 20200021718A1 · Jan 16, 2020