IP Library Granted Patent US 11,551,385
Granted Patent B1
US 11,551,385 · App. 17/356,227 · Granted Jan 10, 2023

Texture replacement system in a multimedia

Inventors: Xibeijia Guan (Singapore, SG); Tiecheng Wu (Singapore, SG); Bo Li (Singapore, SG)
Assignee: Black Sesame Technologies Inc.
G06T11/001G06T7/11G06T7/194G06T7/40G06T2207/10016G06T2207/10024
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,551,385
App. No.
17/356,227
Granted
Jan 10, 2023
Kind
B1
Abstract

The present invention discloses system and method for replacing a texture of a background region in a multimedia. The system is an AI-based multi-media processing system for replacing the original texture of the background of the multimedia with a texture template. The system applies a foreground mask to hide and protect the foreground region and multiple textures of the background. The system uses deep learning to segment specific texture from images or video sequences. The system replaces the texture of the original input image with a texture template to form a processed image.

Claims (31)

1. A system for texture replacement in a multimedia, comprising:

a segmentation module, wherein the segmentation module segments the multimedia to a background region with one or more textures and a foreground region, and compares the one or more textures with pre-defined textures to generate one or more identified textures, further wherein the segmentation module comprising:

a portrait map unit, wherein the portrait: map unit protects the foreground region; and

a texture map unit, wherein the texture map unit replaces the one or more identified textures with a texture template;

a tracking module, wherein the tracking module comprising:

a first tracker unit, wherein the first tracker unit tracks feature matching of the one or more identified textures to guide the texture template; and

a second tracker unit, wherein the second tracker unit tracks movement of the background region and the foreground region, wherein the movement of background region guides a movement of the texture template;

a fusion module, wherein the fusion module adjusts color tone of the texture template based on the multimedia to generate a processed texture; and

a replacement module, wherein the replacement module replaces the one or more textures with the processed texture, and combines the processed texture with the foreground region to form a texture replaced multimedia.

2. The system of claim 1 , wherein the system is configured with an electronic device.

3. The system of claim 2 , wherein the electronic device is a smart phone, a tablet or a camera.

4. The system of claim 2 , wherein the pre-defined textures are stored in a memory of the electronic device.

5. The system of claim 1 , wherein the multimedia is either of an image, a video, an animation.

6. The system of claim 1 , wherein the segmentation module uses artificial intelligence and machine learning algorithm to segment the background section and the foreground section.

7. The system of claim 6 , wherein the segmentation module uses artificial intelligence and machine learning algorithm for comparing the one or more textures with pre-defined textures to generate one or more identified textures.

8. The system of claim 1 , wherein the portrait map unit protects the foreground region by using a portrait mask.

9. The system of claim 1 , wherein the feature matching of the one or more identified textures is based on an optical flow algorithm.

10. The system of claim 9 , wherein the optical flow algorithm determines pattern of apparent motion of objects, surfaces, and edges in the multimedia.

11. The system of claim 1 , wherein the feature matching of the one or more identified textures is based on a feature mapping algorithm.

12. The system of claim 11 , wherein the feature mapping algorithm determines pattern of changing scale, intensity, and rotation.

13. The system of claim 2 , wherein guiding the movement of the texture template is based on sensing the movement by a motion sensor of the electronic device.

14. The system of claim 13 , wherein the motion sensor is an accelerometer or a Gyro-meter.

15. The system of claim 13 , wherein the motion is either of a rotation motion, a translation motion and a scaling motion.

16. The system of claim 1 , wherein the fusion module is based on a Generative adversarial networks (GAN) model.

17. The system of claim 1 , wherein the fusion module comprises an encoder for encoding the one or more identified textures and the template texture to produce the processed texture.

18. The system of claim 1 , wherein the fusion module comprises a decoder for decoding the processed texture to the one or more identified textures.

19. A method for replacing a texture in a multimedia, wherein the method comprising: segmenting one or more textures from a background region and a foreground region, wherein the one or more textures are compared with a plurality of pre-defined textures to generate one or more identified textures;

tracking movement of the foreground region and the pre-defined textures for simulating the texture movement;

adjusting color tone of a texture template to be consistent with at least one of the one or more identified textures, wherein the texture template is retrieved for a texture selected by user from the one or more identified textures to form a processed texture;

replacing the processed texture with the selected texture; and

merging the processed texture with the foreground region forming a texture replaced multimedia.

Assignments (2)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Dec 3, 2021
From: BLACK SESAME INTERNATIONAL HOLDING LIMITED
To: BLACK SESAME TECHNOLOGIES INC.
Reel/Frame 058302/0860 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Oct 26, 2021
From: GUAN, XIBEIJIA; WU, TIECHENG; LI, BO
To: BLACK SESAME INTERNATIONAL HOLDING LIMITED
Reel/Frame 057919/0643 →