IP Library Granted Patent US 12,219,297
Granted Patent B2
US 12,219,297 · App. 17/690,588 · Granted Feb 4, 2025

Image processing system and method

Inventors: Oliver Hume (London, GB); Fabio Cappello (London, GB); Timothy Bradley (London, GB)
Assignee: Sony Interactive Entertainment Inc.
H04N9/643G06T7/174G06T7/90G06V10/22G06V10/56G06V10/70G06V40/15G06V40/16G06T2207/10016G06T2207/10024G06T2207/20081G06T2207/20092G06T2207/30201
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,219,297
App. No.
17/690,588
Granted
Feb 4, 2025
Kind
B2
Abstract

An image processing system for modifying one or more images of a user to change a cardiovascular activity that is measurable from the one or more images, the system comprising: an image receiving section configured to receive image data comprising a plurality of image frames; a region selection section configured to select one or more regions where the colour of the skin of the user changes temporally in the image data, wherein the colour changes are indicative of the cardiovascular activity of the user; and a modifying section configured to modify the colour change of the one or more selected regions in one or more image frames to create modified image data, as described elsewhere herein.

Claims (23)

1. An image processing system for modifying one or more images of a user to change a measurable cardiovascular activity that is measurable from the one or more images, the system comprising:

an image receiving section configured to receive image data comprising a plurality of image frames taken of the user;

a region selection section configured to select one or more regions where temporal changes in a colour of the user's skin are present in the image data of the image frames, wherein the temporal changes in the colour are indicative of the cardiovascular activity of the user; and

a modifying section configured to modify the temporal changes in the colour of the selected one or more regions in one or more of the image frames to create modified image data related to the one or more images of the user such that the modified image data is not indicative of the cardiovascular activity of the user.

2. The image processing system of claim 1 , wherein the modifying section is configured to modify the temporal changes in the colour such that the modified image data is indicative of a predetermined cardiovascular activity representative of a cardiovascular activity.

3. The image processing system of claim 2 , wherein a temporal pattern of the temporal changes in the colour is selected by the user.

4. The image processing system of claim 1 , wherein the modifying section is configured to modify the temporal changes in the colour such that the modified image data is not indicative of cardiovascular activity.

5. The image processing system of claim 4 , in which the modifying section is configured to modify the temporal changes in the colour of the selected one or more regions by randomly perturbing a colour value within a predetermined range for each image frame in the plurality of image frames.

6. The image processing system of claim 1 , wherein the modifying section is configured to modify the temporal changes in the colour of each of the selected one or more regions to a rolling average of colour of each of the selected one or more regions over a predetermined number of preceding image frames.

7. The image processing system of claim 1 , wherein the modifying section is operable to apply a colour change generated by a machine learning model that is trained to control a measurement of the cardiovascular activity of the user, the measurement being generated by an imaging system configured to detect the cardiovascular activity of the user indicated in the image data.

8. The image processing system of claim 7 , wherein the machine learning model is an adversarial machine learning model.

9. The image processing system of claim 1 , wherein the region selection section is configured to select the one or more regions where the temporal changes in the colour of the skin of the user relative to a temporal colour change averaged across all of the skin of the user are present in the image data.

10. The image processing system of claim 1 , wherein the region selection section is configured to select the one or more regions where the temporal changes in the colour of the skin of the user are present over a predetermined number of frames.

11. A camera comprising the image processing system of claim 1 .

12. The image processing system of claim 1 , wherein the modifying section is configured to be activated in response to a predetermined trigger condition.

13. An image processing method for modifying one or more images of a user to change a measurable cardiovascular activity that is measurable from the one or more images, the method comprising the steps of:

receiving image data comprising a plurality of image frames taken of the user;

selecting one or more regions where temporal changes in a colour of the user's skin are present in the image data of the image frames, wherein the temporal changes in the colour are indicative of the cardiovascular activity of the user; and

modifying the temporal changes in the colour of the selected one or more regions in one or more of the image frames to create modified image data related to the one or more images of the user such that the modified image data is not indicative of the cardiovascular activity of the user.

14. A non-transitory machine-readable storage medium which stores computer software which, when executed by a computer, causes the computer to perform a method for modifying one or more images of a user to change a measurable cardiovascular activity that is measurable from the one or more images, the method comprising the steps of:

receiving image data comprising a plurality of image frames taken of the user;

selecting one or more regions where temporal changes in a colour of the user's skin are present in the image data of the image frames, wherein the temporal changes in the colour are indicative of the cardiovascular activity of the user; and

modifying the temporal changes in the colour of the selected one or more regions in one or more of the image frames to create modified image data related to the one or more images of the user such that the modified image data is not indicative of the cardiovascular activity of the user.

Assignments (4)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Apr 28, 2022
From: SONY INTERACTIVE ENTERTAINMENT EUROPE LIMITED
To: SONY INTERACTIVE ENTERTAINMENT INC.
Reel/Frame 059761/0698 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Apr 28, 2022
From: CAPPELLO, FABIO
To: SONY INTERACTIVE ENTERTAINMENT EUROPE LIMITED
Reel/Frame 059820/0812 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Mar 18, 2022
From: BRADLEY, TIMOTHY
To: SONY INTERACTIVE ENTERTAINMENT INC.
Reel/Frame 059305/0095 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Mar 9, 2022
From: HUME, OLIVER
To: SONY INTERACTIVE ENTERTAINMENT INC.
Reel/Frame 059211/0479 →
Priority Claims (1)
GB 2103825 · Mar 19, 2021 · national
Continuity (1)
Related Publication 20220303514A1 · Sep 22, 2022
References Cited (10)
US 20140161421A1 · Shoemaker · 2014 [cited by applicant]
US 20170238842A1 · Jacquel · 2017 [cited by examiner]
US 20190012531A1 · Radwin · 2019 [cited by examiner]
US 20190046056A1 · Khachaturian · 2019 [cited by examiner]
US 20190197368A1 · Madani · 2019 [cited by examiner]
Shamsabadi AS, Sanchez-Matilla R, Cavallaro A. Colorfool: Semantic adversarial colorization. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition 2020 (pp. 1151-1160). (Year: 2020). [cited by examiner]
Extended European Search Report for corresponding EP Application No. 22157063.3, 11 pages, dated Jul. 5, 2022. [cited by applicant]
Combined search and examination report for corresponding GB Application No. GB2103825.2, 13 pages, dated Jan. 6, 2022. [cited by applicant]
Tsou Yun-Yun, et al., “Multi-task Learning for Simultaneous Video Generation and Remote Photoplethysmography Estimation” 15th Asian Conference on Computer Vision, vol. 12626, 16 pages, Nov. 30-Dec. 4, 2020. [cited by applicant]
Communication Pursuant to Article 94(3)EPC for corresponding EP Application No. 22157063.3, 5 pages, dated Mar. 7, 2024. [cited by applicant]