IP Library › Granted Patent US 12,277,688

Granted Patent B2

US 12,277,688 · App. 17/363,253 · Granted Apr 15, 2025

Multi-task text inpainting of digital images

Inventors: Vijay Kumar Baikampady Gopalkrishna (Santa Clara, CA); Raja Bala (Pittsford, NY)

Assignee: CAREAR HOLDINGS LLC

G06T5/77 G06T7/11 G06T7/194 G06T11/60 G06T2207/20081 G06T2207/20084

View Patent ↗

Loading inventors, assignments & file history…

Monitor This Case

Get email alerts when status or documents change.

Order Certified Copies

Most orders are placed with the USPTO same day — all within 24 business hours.

Order via The Patent Place →

Pre-filled with this patent's details

Quick Facts

Patent No.: US 12,277,688
App. No.: 17/363,253
Granted: Apr 15, 2025
Kind: B2

External Links

USPTO Patent Center↗ Full Patent PDF↗

Abstract

A multi-task text infilling system receives a digital image and identifies a region of interest of the image that contains original text. The system uses a machine learning model to determine, in parallel: a foreground image that includes the original text; a background image that omits the original text; and a binary mask that distinguishes foreground pixels from background pixels, The system receives a target mask that contains replacement text. The system then applies the target mask to blend the background image with the foreground layer image and yield a modified digital image that includes the replacement text and omits the original text.

Claims (161)

1. A digital image frame editing method comprising, by a processor:

receiving a digital image frame;

processing the digital image frame to define a region of interest (ROI) that contains original text;

processing the ROI through a multi-task machine learning model to predict, in parallel processes:

a foreground image of the ROI, wherein the foreground image comprises the original text,

a background image of the ROI, wherein the background image omits the original text, and

a binary mask that distinguishes foreground image pixels from background image pixels in the ROI;

receiving a target mask that contains replacement text; and

applying the target mask to blend the background image with the foreground image and yield a modified digital image that includes the replacement text and omits the original text,

wherein the multi-task machine learning model comprises:

a single deep neural encoder that receives the ROI, and

separate deep neural decoders for predicting each of the foreground image, the background image and the binary mask; and

wherein the method further comprises, before applying the target mask:

using the binary mask to extract an average background signal of the ROI, and

using the average background signal to modify the background image produced by the decoder that predicted the background image, wherein using the average background signal to modify the background image comprises:

generating a residual signal as a difference between the average background signal extracted by the binary mask and the average background signal of the predicted background image, and

modifying the predicted background image by adding the residual signal to substantially every pixel in the predicted background image.

2. The method of claim 1 further comprising, before receiving the digital image frame, training the multi-task machine learning model on a dataset comprising ground truth foreground images, ground truth background images and ground truth mask images with one or more of the following losses

background reconstruction loss;

background adversarial loss;

foreground reconstruction loss;

mask reconstruction loss;

end to end reconstruction loss; or

end to end adversarial loss.

3. The method of claim 1 further comprising, before applying the target mask:

using the binary mask to extract an average foreground signal from the foreground image; and

using the average foreground signal to modify the foreground image produced by the decoder that predicted the foreground image.

4. The method of claim 3 , wherein using the average foreground signal to modify the foreground image comprises modifying the foreground image by adding the extracted average foreground signal to the foreground image.

5. The method of claim 1 , wherein applying the target mask to blend the background image with the foreground image and yield the modified digital image further comprises:

generating a modified ROI by applying the target mask to blend the background image with the foreground image; and

replacing the ROI of the received digital image frame with the modified ROI.

6. The method of claim 1 , wherein the blending comprises:

assigning a color value of the foreground image to foreground pixels of the target mask; and

assigning a color value of the background image to background pixels of the target mask.

7. The method of claim 1 , wherein the blending comprises using a generative neural network to blend the foreground image with the background image through the target mask.

8. The method of claim 1 , wherein the machine learning model comprises a deep neural network.

9. The method of claim 1 , wherein processing the digital image frame to define the ROI comprises:

applying a text detector to the digital image frame to return bounding box coordinates; and

defining the ROI according to the bounding box coordinates.

10. A digital image frame editing system comprising:

a processor; and

a memory device containing programming instructions that are configured to cause the processor to:

receive a digital image frame,

process the digital image frame to define a region of interest (ROI) that contains original text,

process the ROI through a multi-task machine learning model to determine, in parallel processes:

a foreground image of the ROI, wherein the foreground image comprises the original text;

a background image of the ROI, wherein the background image omits the original text; and

a binary mask that distinguishes foreground image pixels from background image pixels in the ROI,

use the binary mask to extract an average foreground signal from the foreground image,

modify the foreground image by adding the extracted average foreground signal to the foreground image,

receive a target mask that contains replacement text, and

apply the target mask to blend the background image with the foreground image and yield a modified digital image that includes the replacement text and omits the original text.

11. The system of claim 10 wherein the multi-task machine learning model comprises:

a single deep neural encoder for receiving the ROI; and

separate deep neural decoders for predicting each of the foreground image, the background image and the binary mask.

12. The system of claim 10 further comprising:

a dataset comprising ground truth foreground images, ground truth background images and ground truth mask images; and

additional programming instructions that are configured to cause the processor to, before receiving the digital image frame, train the multi-task machine learning model on the dataset with one or more of the following losses

background reconstruction loss,

background adversarial loss,

foreground reconstruction loss,

mask reconstruction loss,

end to end reconstruction loss, or

end to end adversarial loss.

13. The system of claim 10 , further comprising additional instructions configured to cause the processor to, before applying the target mask:

generate a residual correction signal; and

apply the residual correction signal to the background image.

14. The system of claim 10 , further comprising additional instructions configured to cause the processor to:

use the binary mask to extract an average background signal from the ROI of the digital image frame;

identify an average background signal of the background image;

generate a residual signal as a difference between the average background signal extracted by the binary mask for the ROI of the digital image frame and the average background signal of the background image; and

modify the background image by adding the residual signal to substantially every pixel in the background image.

15. The system of claim 10 , wherein the instructions to apply the target mask to blend the background image with the foreground image and yield the modified digital image further comprise instructions to:

generate a modified ROI by applying the target mask to blend the background image with the foreground image; and

replace the ROI of the received digital image frame with the modified ROI.

16. The system of claim 10 , wherein the instructions to blend comprise instructions to:

assign a color value of the foreground image to foreground pixels of the target mask; and

assign a color value of the background image to background pixels of the target mask.

17. The system of claim 10 , wherein the instructions to blend comprise instructions to use a generative neural network to blend the foreground image with the background image through the target mask.

18. The system of claim 10 , wherein the machine learning model comprises a deep neural network.

19. The system of claim 10 , wherein the instructions to process the digital image frame to define the ROI comprise instructions to:

apply a text detector to the digital image frame to return bounding box coordinates; and

define the ROI according to the bounding box coordinates.

20. A digital image frame editing method comprising, by a processor:

receiving a digital image frame;

processing the digital image frame to define a region of interest (ROI) that contains original text;

processing the ROI through a multi-task machine learning model to predict, in parallel processes:

a foreground image of the ROI, wherein the foreground image comprises the original text,

a background image of the ROI, wherein the background image omits the original text, and

a binary mask that distinguishes foreground image pixels from background image pixels in the ROI;

receiving a target mask that contains replacement text; and

applying the target mask to blend the background image with the foreground image and yield a modified digital image that includes the replacement text and omits the original text,

wherein the multi-task machine learning model comprises:

a single deep neural encoder that receives the ROI, and

separate deep neural decoders for predicting each of the foreground image, the background image and the binary mask, and

wherein the method further comprises, before applying the target mask:

using the binary mask to extract an average foreground signal from the foreground image, and

using the average foreground signal to modify the foreground image produced by the decoder that predicted the foreground image, wherein using the average foreground signal to modify the foreground image comprises modifying the foreground image by adding the extracted average foreground signal to the foreground image.

21. The method of claim 20 further comprising, before receiving the digital image frame, training the multi-task machine learning model on a dataset comprising ground truth foreground images, ground truth background images and ground truth mask images with one or more of the following losses

background reconstruction loss;

background adversarial loss;

foreground reconstruction loss;

mask reconstruction loss;

end to end reconstruction loss; or

end to end adversarial loss.

22. The method of claim 20 further comprising, before applying the target mask:

using the binary mask to extract an average background signal of the ROI; and

using the average background signal to modify the background image produced by the decoder that predicted the background image.

23. The method of claim 20 , wherein applying the target mask to blend the background image with the foreground image and yield the modified digital image further comprises:

generating a modified ROI by applying the target mask to blend the background image with the foreground image; and

replacing the ROI of the received digital image frame with the modified ROI.

24. The method of claim 20 , wherein the blending comprises:

assigning a color value of the foreground image to foreground pixels of the target mask; and

assigning a color value of the background image to background pixels of the target mask.

25. The method of claim 20 , wherein the blending comprises using a generative neural network to blend the foreground image with the background image through the target mask.

26. The method of claim 20 , wherein the machine learning model comprises a deep neural network.

27. The method of claim 20 , wherein processing the digital image frame to define the ROI comprises:

applying a text detector to the digital image frame to return bounding box coordinates; and

defining the ROI according to the bounding box coordinates.

28. A digital image frame editing system comprising:

a processor; and

a memory device containing programming instructions that are configured to cause the processor to:

receive a digital image frame,

process the digital image frame to define a region of interest (ROI) that contains original text,

process the ROI through a multi-task machine learning model to determine, in parallel processes:

a foreground image of the ROI, wherein the foreground image comprises the original text;

a background image of the ROI, wherein the background image omits the original text; and

a binary mask that distinguishes foreground image pixels from background image pixels in the ROI,

use the binary mask to extract an average background signal from the ROI of the digital image

frame,

identify an average background signal of the background image,

modify the background image by adding the residual signal to substantially every pixel in the background image,

receive a target mask that contains replacement text, and

apply the target mask to blend the background image with the foreground image and yield a modified digital image that includes the replacement text and omits the original text.

29. The system of claim 28 , wherein the multi-task machine learning model comprises:

a single deep neural encoder for receiving the ROI; and

separate deep neural decoders for predicting each of the foreground image, the background image and the binary mask.

30. The system of claim 28 , further comprising:

a dataset comprising ground truth foreground images, ground truth background images and ground truth mask images; and

background reconstruction loss,

background adversarial loss,

foreground reconstruction loss,

mask reconstruction loss,

end to end reconstruction loss, or

end to end adversarial loss.

31. The system of claim 28 , further comprising additional instructions configured to cause the processor to, before applying the target mask:

generate a residual correction signal; and

apply the residual correction signal to the background image.

32. The system of claim 28 , wherein the instructions to apply the target mask to blend the background image with the foreground image and yield the modified digital image further comprise instructions to:

generate a modified ROI by applying the target mask to blend the background image with the foreground image; and

replace the ROI of the received digital image frame with the modified ROI.

33. The system of claim 28 , wherein the instructions to blend comprise instructions to:

assign a color value of the foreground image to foreground pixels of the target mask; and

assign a color value of the background image to background pixels of the target mask.

34. The system of claim 28 , wherein the instructions to blend comprise instructions to use a generative neural network to blend the foreground image with the background image through the target mask.

35. The system of claim 28 , wherein the machine learning model comprises a deep neural network.

36. The system of claim 28 , wherein the instructions to process the digital image frame to define the ROI comprise instructions to:

apply a text detector to the digital image frame to return bounding box coordinates; and

define the ROI according to the bounding box coordinates.

Assignments (2)

ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded May 24, 2022

From: PALO ALTO RESEARCH CENTER INCORPORATED

To: CAREAR HOLDINGS LLC

Reel/Frame 060001/0765 →

ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jun 30, 2021

From: GOPALKRISHNA, VIJAY KUMAR BAIKAMPADY; BALA, RAJA

To: PALO ALTO RESEARCH CENTER INCORPORATED

Reel/Frame 056716/0591 →

Continuity (1)

Related Publication 20230005107A1 · Jan 5, 2023

References Cited (41)

US 9014255B2 · Schweid et al. · 2015 [cited by applicant]

US 9042649B2 · Tian · 2015 [cited by examiner]

US 9300840B2 · Hasegawa · 2016 [cited by examiner]

US 10896493B2 · Safdarnejad et al. · 2021 [cited by applicant]

US 20140185104A1 · Hasegawa · 2014 [cited by applicant]

US 20160004921A1 · McCarthy · 2016 [cited by applicant]

US 20160140701A1 · Angara et al. · 2016 [cited by applicant]

US 20200074709A1 · Bhat et al. · 2020 [cited by applicant]

US 20200320324A1 · Goncalves et al. · 2020 [cited by applicant]

US 20210357512A1 · Busila et al. · 2021 [cited by applicant]

US 20220292649A1 · Wang et al. · 2022 [cited by applicant]

US 20220301118A1 · Frey · 2022 [cited by examiner]

DE 102018128531A1 · 2020 [cited by examiner]

WO WO2020038207A1 · 2020 [cited by examiner]

Wu, L. et al., “Editing Text in the Wild”, ACM Int. Conf. Multimedia, 2019. (Year: 2019). [cited by examiner]

Azadi, S. et al., “Multi-content gan for few-shot font style transfer”, IEEE Conf. Comput. Vis. Pattern Recog., 2018. [cited by applicant]

Chen, J. et al., “Bilateral Guided Upsampling”, ACM Trans. Graph., 35(6), 2016. [cited by applicant]

Gharbi, M. et al., Deep bilateral learning for real-time image enhancement, ACM Trans. Graph., 36(4), 2017. [cited by applicant]

Gharbi, M. et al., “Transform Recipes for Efficient Cloud Photo Enhancement”, ACM Trans. Graph., 34(6), 2015. [cited by applicant]

He, K. et al., “Mask R-CNN,” Computer Vision Foundation, pp. 2961-2969. [cited by applicant]

Hu, J. et al., “Image-to-Image Translation with Conditional-GAN,” CS230: Deep Learning, Spring 2018, Stanford University, CA. [cited by applicant]

Kim, D. et al., “Deep Blind Video Decaptioning by Temporal Aggregation and Recurrence,” Computer Vision Foundation, pp. 4263-4272. [cited by applicant]

Liu, G. et al., “Image Inpainting for Irregular Holes Using Partial Convolutions”, ECCV 2018. [cited by applicant]

Nakamura, T., et al., “Scene Text Eraser,” arXiv:1705.02772v1 [cs:CV] May 8, 2017. [cited by applicant]

Roy, P. et al., “STEFANN: Scene Text Editor Using Font Adaptive Neural Network”, IEEE Conf. Comput. Vis. Pattern Recog., 2020. [cited by applicant]

Tursun, O. et al., “MTRNet: A Generic Scene Text Eraser,” arXiv:1903.04092v3 [cs.CV] Oct. 22, 2019. [cited by applicant]

Ulyanov, D., et al., “Deep Image Prior,” Computer Vision Foundation, pp. 9446-9454. [cited by applicant]

Wang, T.C. et al., “High-Resolution Image Synthesis and Semantic Manipulation with Conditional GA Ns,” arXiv:1711.11585v2 [cs.CV] Aug. 20, 2018. [cited by applicant]

Wang, T.C. et al., “Video-to-Video Synthesis,” arXiv:1808.06601v2 [cs:CV] Dec. 3, 2018. [cited by applicant]

Wexler, Y. et al., “Space-Time Video Completion,” Dept. of Computer Science and Applied Math, The Weizmann Institute of Science, Rehovot, 76100 Israel. [cited by applicant]

Wu, L. et al., “Editing Text in the Wild”, ACM Int. Conf. Multimedia, 2019. [cited by applicant]

Xu, R. et al., “Deep Flow-Guided Video Inpainting,” Computer Vision Foundation, pp. 3723-3732. [cited by applicant]

Yang, Q. et al., “Swaptext: Image based texts transfer in scenes”, IEEE Conf. Comput. Vis. Pattern Recog., 2020. [cited by applicant]

Yang, S. et al. “Controllable Artistic Text Style Transfer via Shape-Matching GAN”, Int. Conf. Comput. Vis., 2019. [cited by applicant]

Yang, S. et al., “Awesome typography: Statistics-based text effects transfer” IEEE Conf. Comput. Vis. Pattern Recog., 2017. [cited by applicant]

Yang, S. et al., “TET-GAN: Text Effects Transfer via Stylization and Destylization”, AAAI 2019. [cited by applicant]

Yu, J. et al., “Free-Form Image Inpainting with Gated Convolution”, ICCV, arXiv:1806.03589 [cs.CV], 2018. [cited by applicant]

Yu, J. et al., “Generative Image Inpainting with Contextual Attention,” Computer Vision Foundation, pp. 5505-5514. [cited by applicant]

Zhang, L. et al., “Deep Image Blending”, WAVC 2020 paper, IEEE Xplore. [cited by applicant]

Zhang, S. et al., “EnsNet: Ensconce Text in the Wild,” arXiv:1812.00723v1 [cs.CV] Dec. 3, 2018. [cited by applicant]

Information about Related Patents and Patent Applications, see section 4 of the accompanying Information Disclosure Statement Letter, which concerns Related Patents and Patent Applications. [cited by applicant]