IP Library › Granted Patent US 11,294,422
Granted Patent B1
US 11,294,422 · App. 16/580,477 · Granted Apr 5, 2022

Electronic device including a camera disposed behind a display

Inventors: Manohar B. Srikanth (Mountain View, CA); Marduke Yousefpor (San Jose, CA); Ting Sun (Cupertino, CA); Kathrin Berkner-Cieslicki (Los Altos, CA); Mohammad Yeke Yazdandoost (San Jose, CA); Ricardo J. Motta (Palo Alto, CA)
Assignee: APPLE INC.
G06F1/1605G06F1/1626G06F1/1637H04N5/23293
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,294,422
App. No.
16/580,477
Granted
Apr 5, 2022
Kind
B1
Abstract

In one implementation, an apparatus includes a display having a front surface and a back surface. The display includes a plurality of pixel regions that emit light from the front surface to display a displayed image and a plurality of apertures that transmit light from the front surface to the back surface. The apparatus includes a camera disposed on a side of the back surface of the display. The camera is configured to capture a captured image. The apparatus includes a processor coupled to the display and the camera. The processor is configured to receive the captured image and apply a first digital filter to a first portion of the captured image and a second digital filter, different than the first digital filter, to a second portion of the captured image to reduce image distortion caused by the display.

Claims (45)

1. An apparatus comprising:

a display having a front surface and a back surface, the display including a plurality of pixel regions that emit light from the front surface to display a displayed image and a plurality of apertures that transmit light from the front surface to the back surface;

a camera disposed on a side of the back surface of the display, the camera configured to capture a captured image; and

a processor coupled to the display and the camera, the processor configured to receive the captured image and apply a first digital filter to a first portion of the captured image and a second digital filter, different than the first digital filter, to a second portion of the captured image to reduce image distortion caused by the display.

2. The apparatus of claim 1 , wherein the first portion of the captured image corresponds to a first color pane of the captured image and the second portion of the captured image corresponds to a second color pane, different than the first color pane, of the captured image.

3. The apparatus of claim 2 , wherein the processor is further configured to demosaic the captured image after applying the first digital filter and the second digital filter.

4. The apparatus of claim 2 , wherein the processor is further configured to demosaic the captured image after applying the first digital filter and the second digital filter.

5. The apparatus of claim 1 , wherein the first portion of the captured image corresponds to a first spatial portion of the captured image and the second portion of the captured image corresponds to a second spatial portion of the captured image.

6. The apparatus of claim 5 , wherein the first spatial portion is of a different size than the second spatial portion.

7. The apparatus of claim 5 , wherein the first spatial portion is irregularly shaped.

8. The apparatus of claim 1 , wherein the processor is further configured to:

estimate a point spread function of the display; and

generate the first digital filter based on the point spread function; and

generate the second digital filter, different from the first digital filter, based on the point spread function.

9. The apparatus of claim 1 , wherein the processor is further configured to:

estimate a first point spread function of a first portion of the display corresponding to the first portion of the captured image and estimate a second point spread function, different from the first point spread function, of a second portion of the display corresponding to the second portion of the captured image; and

generate the first digital filter based on the first point spread function and generating the second digital filter, different from the first digital filter, based on the second point spread function.

10. The apparatus of claim 1 , wherein the processor is configured to apply the first digital filter and apply the second digital filter by:

partitioning the captured image into the first portion and the second portion;

applying the first digital filter to the first portion to generate a filtered first portion;

applying the second digital filter to the second portion to generate a filtered second portion; and

assembling the filtered first portion and the filtered second portion to generate a filtered captured image.

11. A method comprising:

receiving, from a camera disposed on a side of a back surface of a display that emits light from a front surface of the display, a captured image;

applying a first digital filter to a first portion of the captured image and applying a second digital filter, different than the first digital filter, to a second portion of the captured image to reduce image distortion caused by the display.

12. The method of claim 11 , wherein the first portion of the captured image corresponds to a first color pane of the captured image and the second portion of the captured image corresponds to a second color pane, different than the first color pane, of the captured image.

13. The method of claim 11 , wherein the first portion of the captured image corresponds to a first spatial portion of the captured image and the second portion of the captured image corresponds to a second spatial portion of the captured image.

14. The method of claim 13 , wherein the first spatial portion is of a different size than the second spatial portion.

15. The method of claim 11 , further comprising:

estimating a point spread function of the display; and

generating the first digital filter based on the point spread function; and

generating the second digital filter, different from the first digital filter, based on the point spread function.

16. The method of claim 11 , further comprising:

estimating a first point spread function of a first portion of the display corresponding to the first portion of the captured image and estimating a second point spread function, different from the first point spread function, of a second portion of the display corresponding to the second portion of the captured image; and

generating the first digital filter based on the first point spread function and generating the second digital filter, different from the first digital filter, based on the second point spread function.

17. The method of claim 11 , wherein applying the first digital filter and applying the second digital filter comprises:

partitioning the captured image into the first portion and the second portion;

applying the first digital filter to the first portion to generate a filtered first portion;

applying the second digital filter to the second portion to generate a filtered second portion; and

assembling the filtered first portion and the filtered second portion to generate a filtered captured image.

18. A non-transitory computer-readable medium having instructions encoded thereon which, when executed by a processor of a device, cause the device to:

receive, from a camera disposed on a side of a back surface of a display that emits light from a front surface of the display, a captured image;

apply a first digital filter to a first portion of the captured image and applying a second digital filter, different than the first digital filter, to a second portion of the captured image to reduce image distortion caused by the display.

19. The non-transitory computer-readable medium of claim 18 , wherein the first portion of the captured image corresponds to a first color pane of the captured image and the second portion of the captured image corresponds to a second color pane, different than the first color pane, of the captured image.

20. The non-transitory computer-readable medium of claim 18 , wherein the first portion of the captured image corresponds to a first spatial portion of the captured image and the second portion of the captured image corresponds to a second spatial portion of the captured image.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Oct 31, 2019
From: SRIKANTH, MANOHAR B.; YOUSEFPOR, MARDUKE; SUN, TING; BERKNER-CIESLICKI, KATHRIN; YAZDANDOOST, MOHAMMAD YEKE; MOTTA, RICARDO J.
To: APPLE INC.
Reel/Frame 050885/0206 →
Continuity (1)
Provisional Application 62737487 · Sep 27, 2018
Cited By (2)
US 12,380,665 US 12,639,867