IP Library Granted Patent US 12,003,853
Granted Patent B2
US 12,003,853 · App. 17/000,098 · Granted Jun 4, 2024

Systems and methods for adjusting focus based on focus target information

Inventors: William Guie Rivard (Menlo Park, CA); Brian J. Kindle (Sunnyvale, CA); Adam Barry Feder (Mountain View, CA)
Assignee: DUELIGHT LLC
H04N23/67H04N5/265H04N23/45H04N23/51H04N23/54H04N23/55H04N23/57H04N23/611H04N23/62H04N23/64H04N23/743H04N23/56H04N23/951
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,003,853
App. No.
17/000,098
Granted
Jun 4, 2024
Kind
B2
Abstract

A system, method, and computer program product are provided for generating a focus sweep to produce a focus stack. In use, an image is sampled as image data. Next, a first focus region is identified and a second focus region is identified. Next, first focus target information corresponding to the first focus region is determined and second focus target information corresponding to the second focus region is determined. Further, a focus is adjusted, based on the first focus target information and at least one first image is captured based on the first focus target information. Additionally, the focus is adjusted, based on the second focus target information and at least one second image is captured based on the second focus target information. Lastly, the at least one first image and the at least one second image are saved to an image stack. Additional systems, methods, and computer program products are also presented.

Claims (113)

1. A device, comprising:

a non-transitory memory storing instructions; and

one or more processors in communication with the non-transitory memory, wherein the one or more processors execute the instructions to:

sample an initial image as image data, using a camera module;

identify a first focus region, using an image classifier, wherein the image classifier identifies or tracks a first set of objects within a first optical scene information;

identify a second focus region, using the image classifier, wherein the image classifier identifies or tracks a second set of objects within a second optical scene information;

determine a first focus target information corresponding to the first focus region;

transmit the first focus target information to a controller, using a camera interconnect connected to the controller and to an application processor, to sample, while an identification of at least one additional focus region is at least initiated, another image, based on the first focus target information that is determined based on the first focus region, which is identified based on the first set of objects in the first image;

determine a second focus target information corresponding to the second focus region; and

transmit the second focus target information to the controller, using the camera interconnect connected to the controller and to the application processor, to sample, while an identification of at least one yet additional focus region or a determination of yet additional focus target information is at least initiated, yet another image based on the second focus target information that is determined based on the second focus region, which is identified based on the second set of objects in the second image.

2. A device, comprising:

a camera module;

a display;

one or more processors; and

a non-transitory memory in communication with the camera module, the display, and the one or more processors, the non-transitory memory storing instructions that, when executed, cause the device to:

sample a first image utilizing the camera module;

identify a first focus region based on a first set of one or more objects in the first image;

determine first focus target information based on the first focus region; and

while at least one of: an identification of at least one additional focus region or a determination of additional focus target information is at least initiated, sample a second image utilizing the camera module, based on the first focus target information that is determined based on the first focus region, which is identified based on the first set of one or more objects in the first image.

3. The device of claim 2 , wherein the device is configured such that the first focus region is automatically identified based on the first set of one or more objects, which are automatically identified in the first image.

4. The device of claim 2 , wherein the device is configured such that the first focus region is automatically identified based on the first set of one or more objects, which are detected to be manually identified in the first image.

5. The device of claim 2 , wherein the device is configured such that the first set of one or more objects is identified in the first image based on a classification algorithm for automatically classifying objects to identify the first set of one or more objects.

6. The device of claim 2 , wherein the device is configured such that the first focus region includes a first boundary.

7. The device of claim 2 , wherein the device is configured such that the first focus target information includes first pixel location information.

8. The device of claim 2 , wherein the instructions, when executed, further cause the device to:

identify a second focus region based on the first set of one or more objects in the second image;

determine second focus target information based on the second focus region; and

sample a third image, based on the second focus target information that is determined based on the second focus region, which is identified based on the first set of one or more objects in the second image.

9. The device of claim 8 , wherein the device is configured such that updated focus regions are automatically and repeatedly identified as the first set of one or more objects is detected as moving, and the camera module automatically and repeatedly focuses on the first set of one or more objects based on the updated focus regions while the first set of one or more objects remains visible.

10. The device of claim 2 , wherein the instructions, when executed, further cause the device to:

identify a second focus region based on a second set of one or more objects in the second image;

determine second focus target information based on the second focus region; and

sample a third image, based on the second focus target information that is determined based on the second focus region, which is identified based on the second set of one or more objects in the second image.

11. The device of claim 2 , wherein the device is configured such that the at least one of: the identification of the at least one additional focus region or the determination of the additional focus target information, includes the identification of the at least one additional focus region.

12. The device of claim 2 , wherein the device is configured such that the at least one of: the identification of the at least one additional focus region or the determination of the additional focus target information, includes the determination of the additional focus target information.

13. The device of claim 2 , wherein the device is configured such that a first circuit of the camera module performs an image sampling process that is capable of sampling the first image and the second image, and a second circuit performs a focus region identification and focus target information determination process that is capable of identifying the first focus region and determining the first focus target information, where the first circuit and the second circuit operate concurrently, at least in part, for increasing a speed with which the device remains focused on the first set of one or more objects.

14. The device of claim 2 , wherein the device is configured such that a first circuit of the camera module performs an image sampling process that is capable of sampling the first image and the second image, and a second circuit performs a focus region identification and focus target information determination process that is capable of identifying the first focus region and determining the first focus target information, where the first circuit and the second circuit are integrated in separate chips.

15. The device of claim 2 , wherein the device is configured such that a first circuit of the camera module performs an image sampling process that is capable of sampling the first image and the second image, and a second circuit performs a focus region identification and focus target information determination process that is capable of identifying the first focus region and determining the first focus target information, where the first circuit and the second circuit are integrated in a single chip.

16. The device of claim 2 , wherein the device is configured such that a first circuit of the camera module performs an image sampling process that is capable of sampling the first image and the second image, and a second circuit performs a focus region identification and focus target information determination process that is capable of identifying the first focus region and determining the first focus target information, where the second circuit continuously performs the focus region identification and focus target information determination process while the first circuit performs the image sampling process.

17. The device of claim 2 , wherein the device is configured such that a first circuit of the camera module performs an image sampling process that is capable of sampling the first image and the second image, and a second circuit performs a focus region identification and focus target information determination process that is capable of identifying the first focus region and determining the first focus target information, where the first circuit continuously performs the image sampling process while the second circuit continuously performs the focus region identification and focus target information determination process.

18. The device of claim 2 , wherein the device is configured such that a first circuit of the camera module performs an image sampling process that is capable of sampling the first image and the second image, and a second circuit performs a focus region identification and focus target information determination process that is capable of identifying the first focus region and determining the first focus target information, where the first circuit performs the image sampling process while the second circuit performs the focus region identification and focus target information determination process, such that at least one intermediary image is sampled: after the first image is sampled and before the second image is sampled, where the at least one additional focus region is identified based on at least one object in the at least one intermediary image.

19. The device of claim 2 , wherein the device is configured such that a first circuit of the camera module performs an image sampling process that is capable of sampling the first image and the second image, and a second circuit performs a focus region identification and focus target information determination process that is capable of identifying the first focus region and determining the first focus target information, where the first circuit performs the image sampling process while the second circuit performs the focus region identification and focus target information determination process, such that at least one intermediary image is sampled: after the first image is sampled and before the second image is sampled, where the additional focus target information is determined based on the at least one additional focus region, which is identified based on at least one object in the at least one intermediary image.

20. The device of claim 2 , wherein the device is configured such that a first circuit of the camera module performs an image sampling process that is capable of sampling the first image and the second image, and a second circuit performs a focus region identification and focus target information determination process that is capable of identifying the first focus region and determining the first focus target information, where the first circuit performs the image sampling process while the second circuit performs the focus region identification and focus target information determination process, such that no intermediary image is sampled: after the first image is sampled and before the second image is sampled, where the at least one additional focus region is identified based on the first set of one or more objects in the first image.

21. The device of claim 2 , wherein the device is configured such that a first circuit of the camera module performs an image sampling process that is capable of sampling the first image and the second image, and a second circuit performs a focus region identification and focus target information determination process that is capable of identifying the first focus region and determining the first focus target information, where the first circuit performs the image sampling process independent of the second circuit performing the focus region identification and focus target information determination process.

22. The device of claim 2 , wherein the device is configured such that a first circuit of the camera module performs an image sampling process that is capable of sampling the first image and the second image, and a second circuit performs a focus region identification and focus target information determination process that is capable of identifying the first focus region and determining the first focus target information, such that the image sampling process and the focus region identification and focus target information determination process occur concurrently.

23. The device of claim 2 , wherein the device is configured such that a first circuit of the camera module performs an image sampling process that is capable of sampling the first image and the second image, and a second circuit performs a focus region identification and focus target information determination process that is capable of identifying the first focus region and determining the first focus target information, such that the image sampling process and the focus region identification and focus target information determination process occur concurrently so that at least one image is sampled concurrently with at least one of: an identification of at least one focus region, or a generation of at least one focus target information.

24. The device of claim 2 , wherein the device is configured such that a first circuit of the camera module performs an image sampling process that is capable of sampling the first image and the second image, and a second circuit performs a focus region identification and focus target information determination process that is capable of identifying the first focus region and determining the first focus target information, such that the image sampling process and the focus region identification and focus target information determination process occur concurrently so that at least one image is sampled concurrently with both an identification of at least one focus region, and a generation of at least one focus target information.

25. The device of claim 24 , wherein at least one of:

the non-transitory memory is in direct communication with the camera module, the display, and the one or more processors;

the non-transitory memory is in direct communication with the camera module, the display, and the one or more processors, with no intermediary components therebetween;

the non-transitory memory is in indirect communication with the camera module, the display, and the one or more processors;

the non-transitory memory is in indirect communication with the camera module, the display, and the one or more processors, with at least one intermediary component therebetween;

the instructions are executed by the one or more processors;

the instructions are executed by one or more other processors;

the first circuit and the second circuit are integrated on a same, single chip;

the first circuit and the second circuit are integrated on separate chips;

the first circuit and the second circuit are part of the one or more processors;

the first circuit and the second circuit are not part of the one or more processors;

one or more of the sampling of the first image, identification, generation, sampling of the second image is caused to be performed by a first one of the one or more processors, and one or more other of the sampling of the first image, identification, generation, sampling of the second image is caused to be performed by a second one of the processors;

one or more of the sampling of the first image, identification, generation, sampling of the second image is caused to be performed by the one or more processors, and one or more other of the sampling of the first image, identification, generation, sampling of the second image is caused to be performed by one or more other processors;

all of the sampling of the first image, identification, generation, and sampling of the second image are caused to be performed by a single processor;

the instructions are part of an operating system;

the instructions are part of at least one application program;

the device is caused by an operating system command;

the device is caused by an application program command;

the device is caused by a hardware command;

the first set of one or more objects includes a single object;

the first set of one or more objects includes a plurality of objects;

the first focus target information includes at least one of: one or more coordinates, one or more values, or one or more data;

the first focus target information takes the form of a command;

the first focus target information takes the form of data;

the at least one of: the identification of the at least one additional focus region or the determination of the additional focus target information, includes only the identification of the at least one additional focus region;

the at least one of: the identification of the at least one additional focus region or the determination of the additional focus target information, includes only the determination of the additional focus target information;

the at least one of: the identification of the at least one additional focus region or the determination of the additional focus target information, includes both the identification of the at least one additional focus region and the determination of the additional focus target information;

the while the at least one of: the identification of the at least one additional focus region or the determination of the additional focus target information is at least initiated, includes a time period when the at least one of: the identification of the at least one additional focus region or the determination of the additional focus target information is at least initiated;

the while the at least one of: the identification of the at least one additional focus region or the determination of the additional focus target information is at least initiated, includes a time period after the at least one of: the identification of the at least one additional focus region or the determination of the additional focus target information is at least initiated;

the while the at least one of: the identification of the at least one additional focus region or the determination of the additional focus target information is at least initiated, includes a time period before the at least one of: the identification of the at least one additional focus region or the determination of the additional focus target information is completed;

the image sampling process is further capable of sampling images other than the first image and the second image, and the focus region identification and focus target information determination process is further capable of identifying focus regions other than the first focus region and determining focus target information other than the first focus target information;

the image sampling process is further capable of sampling images other than the first image and the second image, including the at least one image;

the focus region identification and focus target information determination process is further capable of identifying focus regions other than the first focus region, including the at least one focus region;

the focus region identification and focus target information determination process is further capable of determining focus target information other than the first focus target information, including the at least one focus target information;

the image sampling process is further capable of sampling images other than the first image and the second image, including an intermediary image;

the focus region identification and focus target information determination process is further capable of identifying focus regions other than the first focus region, including the at least one focus region which is the at least one additional focus region;

the focus region identification and focus target information determination process is further capable of determining focus target information other than the first focus target information, including the at least one focus target information which is the additional focus target information;

the focus region identification and focus target information determination process is further capable of identifying focus regions other than the first focus region, including the at least one focus region which is different from the at least one additional focus region;

the focus region identification and focus target information determination process is further capable of determining focus target information other than the first focus target information, including the at least one focus target information which is different from the additional focus target information; or

the image sampling process is further capable of sampling images other than the first image and the second image, including the at least one image, and the focus region identification and focus target information determination process is further capable of identifying focus regions other than the first focus region, including the at least one focus region, such that the concurrently refers to the at least one image being sampled concurrently with the at least one focus region.

26. The device of claim 2 , wherein the device is configured such that:

the first focus region is automatically identified based on the first set of one or more objects that are automatically identified in the first image;

the instructions, when executed, further cause the device to:

identify a second focus region based on the first set of one or more objects in the second image,

determine second focus target information based on the second focus region, and

sample a third image, based on the second focus target information that is determined based on the second focus region, which is identified based on the first set of one or more objects in the second image;

updated focus regions are automatically and repeatedly identified as the first set of one or more objects is detected as moving, and the camera module automatically and repeatedly focuses on the first set of one or more objects based on the updated focus regions; and

an image sampling process is capable of sampling the first image and the second image, and a focus region identification and focus target information determination process is capable of identifying the first focus region and determining the first focus target information, where the image sampling process and the focus region identification and focus target information determination process occur concurrently, at least in part, for increasing a speed with which the device focuses on at least one object.

27. The device of claim 26 , wherein the device is configured such that:

the image sampling process and the focus region identification and focus target information determination process are performed continuously while focusing on the at least one object; and

at least one intermediary image is sampled: after the first image is sampled and before the second image is sampled, where the at least one additional focus region is identified based on at least one object in the at least one intermediary image.

28. A method, comprising:

at a device with a display and a camera module:

sampling a first image utilizing the camera module;

identifying a first focus region based on a first set of one or more objects in the first image;

determining a first focus target information based on the first focus region; and

while at least one of: an identification of at least one additional focus region or a determination of additional focus target information is at least initiated, sampling a second image utilizing the camera module, based on the first focus target information that is determined based on the first focus region, which is identified based on the first set of one or more objects in the first image.

29. The method of claim 28 , wherein:

the first focus region is automatically identified based on the first set of one or more objects that are automatically identified in the first image;

and further comprising:

identifying a second focus region based on the first set of one or more objects in the second image,

determining second focus target information based on the second focus region, and

sampling a third image, based on the second focus target information that is determined based on the second focus region, which is identified based on the first set of one or more objects in the second image, wherein:

updated focus regions are automatically and repeatedly identified as the first set of one or more objects is detected as moving, and the camera module automatically and repeatedly focuses on the first set of one or more objects based on the updated focus regions;

an image sampling process samples the first image and the second image, and a focus region identification and focus target information determination process identifies the first focus region and determining the first focus target information, where the image sampling process and the focus region identification and focus target information determination process occur concurrently, at least in part, for increasing a speed with which the device remains focused on the first set of one or more objects; and

at least one intermediary image is sampled: after the first image is sampled and before the second image is sampled, where the at least one additional focus region is identified based on at least one object in the at least one intermediary image.

Assignments (4)
CORRECTIVE ASSIGNMENT TO CORRECT THE LAST NAME OF SECOND INVENTOR FROM "KIINDLE" TO "KINDLE" PREVIOUSLY RECORDED ON REEL 69381 FRAME 342. ASSIGNOR(S) HEREBY CONFIRMS THE CONFIRMATORY ASSIGNMENT OF PATENT RIGHTS. Recorded Jan 28, 2025
From: FEDER, ADAM BARRY; KINDLE, BRIAN; RIVARD, WILLIAM
To: DUELIGHT LLC
Reel/Frame 070039/0172 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Nov 12, 2024
From: FEDER, ADAM BARRY; KIINDLE, BRIAN; RIVARD, WILLIAM
To: DUELIGHT LLC
Reel/Frame 069381/0342 →
SECURITY INTEREST Recorded Jul 30, 2021
From: DUELIGHT, LLC
To: ZILKA KOTAB, P.C.
Reel/Frame 057043/0301 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Aug 25, 2020
From: RIVARD, WILLIAM GUIE; KINDLE, BRIAN J.; FEDER, ADAM BARRY
To: DUELIGHT LLC
Reel/Frame 053593/0760 →
Continuity (4)
Continuation 16213041 · Dec 7, 2018
Continuation 15643311 · Jul 6, 2017
Continuation 15254964 · Sep 1, 2016
Related Publication 20210037178A1 · Feb 4, 2021
Cited By (7)
US 12,242,402 US 12,401,911 US 12,401,912 US 12,418,727 US 12,445,736 US 12,597,221 US 12,666,159