IP Library Granted Patent US 12,219,119
Granted Patent B2
US 12,219,119 · App. 18/399,923 · Granted Feb 4, 2025

Time-of-flight camera system

Inventors: Daniel Van Nieuwenhove (Hofstade, BE); Julien Thollot (Woluwe-Saint-Lambert, BE)
Assignee: Sony Depthsensing Solutions SA/NV
H04N13/25G01S17/86G01S17/894H04N13/122H04N13/128H04N13/207H04N13/243H04N13/246H04N13/257H04N13/271H04N25/41H04N2013/0081
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,219,119
App. No.
18/399,923
Filed
Dec 29, 2023
Granted
Feb 4, 2025
Kind
B2
Art Unit
2488
USPC
348/48
Abstract

The invention relates to a TOF camera system comprising several cameras, at least one of the cameras being a TOF camera, wherein the cameras are assembled on a common substrate and are imaging the same scene simultaneously and wherein at least two cameras are driven by different driving parameters.

Claims (29)

1. A camera system comprising:

at least one first camera, the first camera being a Time-of-Flight camera configured to obtain depthmap information of a scene; and

a second camera, the second camera being an imaging camera, the at least one first camera and second cameras comprising respective first lens and second lens and respective first image sensor and second image sensors,

wherein the at least one first camera and the second camera are arranged on a common support, and

wherein a resolution of an image obtained by the second camera is different than the resolution of the depthmap information obtained by the first camera, the camera system comprising circuitry configured to perform fusion of the depthmap information and the image to form a computed output image.

2. The camera system as claimed in claim 1 , wherein the common support forms a plane on which the first and second image sensors of the first and second camera are arranged.

3. The camera system as claimed in claim 1 wherein the first and second image sensors are in a same imaging plane.

4. The camera system as claimed in claim 1 , wherein the common support is a silicon substrate.

5. The camera system as claimed in claim 1 , wherein camera system processing circuitry is located between the at least one first camera and the second camera on the common support.

6. The camera system as claimed in claim 1 , wherein at least one of the first image sensor and the second image sensor comprise nanoparticles of semiconductor particles configured to control properties of received light.

7. The camera system as claimed in claim 1 , wherein at least one of the first image sensor and the second image sensor comprise Quantum Dot films.

8. The camera system as claimed in claim 1 , wherein the at least one second camera is an RGB-IR camera.

9. The camera system as claimed in claim 1 , wherein another of the at least one first camera is a Time-of-Flight camera comprising a source of modulated illuminating light.

10. The camera system as claimed in claim 1 , wherein the second image sensor comprises pixels configured to respectively receive, via pass filters, red light, green light, blue light and Infra-Red light.

11. The camera system as claimed in claim 1 , wherein the circuitry performs fusion of the depthmap and image from the at least one first camera and second camera respectively, the at least one first camera and second camera both operating with different integration times, one of the integration times providing increased robustness to high motion.

12. The camera system as claimed in claim 1 , wherein a resolution of an image obtained by the second camera is higher than the resolution of the depthmap information obtained by the first camera.

13. The camera system as claimed in claim 1 , wherein the first lens and second lens have different focal lengths.

14. The camera system as claimed in claim 1 , wherein the fusion is performed by the circuitry, by generating a point-by-point correspondence of the depthmap to the image on a common co-ordinate system.

15. A method comprising:

obtaining depthmap information of a scene via least one first camera, the first camera being a Time-of-Flight camera; and

obtaining an image via a second camera wherein a resolution of the image obtained by the second camera is different than the resolution of the depthmap information obtained by the first camera, computing an output image by fusion of the depthmap information and the image, wherein

the second camera is an imaging camera, the at least one first camera and second cameras comprising respective first lens and second lens and respective first image sensor and second image sensors, and wherein the at least one first camera and the second camera are arranged on a common support.

16. A camera system comprising:

at least one first camera, the first camera being a combined color and infrared camera configured to obtain depth information of a scene; and

a second camera, the second camera being an imaging camera, the at least one first camera and second cameras comprising respective first lens and second lens and respective first image sensor and second image sensors,

wherein the at least one first camera and the second camera are arranged on a common support, and

wherein a resolution of an image obtained by the second camera is different than the resolution of the combined color and depth information obtained by the first camera, the camera system comprising circuitry configured to perform fusion of the combined color and depth information and the image obtained by the second camera to form a computed output image.

17. The camera system as claimed in claim 16 , wherein the at least one first camera comprises pass filters for red light, green light, blue light and Infra-Red light and circuitry configured to fuse light properties obtained via the pass filters to provide combined color and depth information.

18. The Camera system as claim in claim 16 , in which the at least one first camera comprises two RGB-Infrared cameras.

Assignments (3)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Apr 29, 2025
From: SONY DEPTHSENSING SOLUTIONS SA/NV
To: SONY SEMICONDUCTOR SOLUTIONS CORPORATION
Reel/Frame 071113/0499 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Mar 14, 2024
From: VAN NIEUWENHOVE, DANIEL; THOLLOT, JULIEN
To: SOFTKINETIC SENSORS NV
Reel/Frame 066767/0077 →
CHANGE OF NAME Recorded Mar 14, 2024
From: SOFTKINETIC SENSORS NV
To: SONY DEPTHSENSING SOLUTIONS SA/NV
Reel/Frame 066767/0089 →
Priority Claims (1)
EP 13199564 · Dec 24, 2013 · regional
Continuity (5)
Continuation 17522863 · Nov 9, 2021
Continuation 16843346 · Apr 8, 2020
Continuation 16460049 · Jul 2, 2019
Division 14904554
Related Publication 20240146895A1 · May 2, 2024
References Cited (49)
US 7791715B1 · Bamji · 2010 [cited by examiner]
US 7994465B1 · Bamji · 2011 [cited by examiner]
US 10397552B2 · Van Nieuwenhove et al. · 2019 [cited by applicant]
US 10638118B2 · Van Nieuwenhove et al. · 2020 [cited by applicant]
US 11172186B2 · Van Nieuwenhove et al. · 2021 [cited by applicant]
US 11863734B2 · Van Nieuwenhove · 2024 [cited by examiner]
US 20100322477A1 · Schmitt · 2010 [cited by applicant]
US 20110025843A1 · Oggier · 2011 [cited by applicant]
US 20110304841A1 · Bamji · 2011 [cited by examiner]
US 20120044093A1 · Pala · 2012 [cited by applicant]
US 20120249744A1 · Pesach · 2012 [cited by applicant]
US 20130188022A1 · Katz · 2013 [cited by applicant]
US 20140028804A1 · Usuda et al. · 2014 [cited by applicant]
US 20140049783A1 · Royo Royo · 2014 [cited by applicant]
US 20140327741A1 · Fleischmann · 2014 [cited by applicant]
US 20150075067A1 · Stowe · 2015 [cited by applicant]
US 20160005179A1 · Petyushko · 2016 [cited by applicant]
US 20160117829A1 · Yoon · 2016 [cited by applicant]
US 20160295193A1 · Van Nieuwenhove et al. · 2016 [cited by applicant]
US 20170134717A1 · Trail · 2017 [cited by examiner]
US 20190291720A1 · Xiao · 2019 [cited by examiner]
US 20200007850A1 · Van Nieuwenhove et al. · 2020 [cited by applicant]
US 20200236342A1 · Van Nieuwenhove et al. · 2020 [cited by applicant]
US 20220070436A1 · Van Nieuwenhove et al. · 2022 [cited by applicant]
CN 101094320A · 2007 [cited by applicant]
CN 101321302A · 2008 [cited by applicant]
EP 2026591A2 · 2009 [cited by applicant]
JP 2002500367A · 2002 [cited by applicant]
JP 2008008687A · 2008 [cited by applicant]
JP 2008164367A · 2008 [cited by applicant]
JP 2008286527A · 2008 [cited by applicant]
JP 2009047497A · 2009 [cited by applicant]
JP 2009139995A · 2009 [cited by applicant]
JP 2013538342A · 2013 [cited by applicant]
KR 1020120056668 · 2012 [cited by applicant]
WO WO9934235A1 · 1999 [cited by applicant]
WO WO2010139628A1 · 2010 [cited by applicant]
WO WO2012012607A2 · 2012 [cited by applicant]
WO WO2012137434A1 · 2012 [cited by applicant]
U.S. Appl. No. 17/522,863, filed Nov. 9, 2021, Van Nieuwenhove et al. [cited by applicant]
EP 13199564.9, Jan. 13, 2017, European Communication. [cited by applicant]
CN 201480048199, Oct. 25, 2017, Office Action. [cited by applicant]
JP 2016-526665, Oct. 24, 2019, Office Action. [cited by applicant]
EP 13199564.9, May 11, 2020, European communication. [cited by applicant]
Chinese Office Action and English Translation thereof for Chinese Application No. 2014800408199 dated Oct. 25, 2017. [cited by applicant]
European Communication for European Application No. 13199564.9 dated Jan. 13, 2017. [cited by applicant]
European Communication in connection with European Application No. 13199564.9, dated May 11, 2020. [cited by applicant]
Japanese Office Action and English Translation thereof for Japanese Application No. 2016-526665 dated Oct. 24, 2017. [cited by applicant]
Huhle et al, On-the-Fly Scene Acquisition with a Handy Multisensor-System. Int. J. of Intelligent Systems Technologies and Applications. Jan. 1, 2007. pp. 1-9, XP055115315. [cited by applicant]