IP Library › Granted Patent US 10,424,118
Granted Patent B2
US 10,424,118 · App. 15/390,827 · Granted Sep 24, 2019

Perspective representation of a virtual scene component

Inventors: Thilo Hannemann (Erlangen, DE); Michael Suehling (Erlangen, DE); Michael Wels (Bamberg, DE); Andreas Wimmer (Forchheim, DE); Ferdinand Distler (Erlangen, DE)
Assignee: Siemens Healthcare GmbH
G06T19/006A61B5/00A61B6/032A61B6/488G06T7/13G06T15/04G06T17/20A61B6/5247G06T2207/10028G06T2207/10048G06T2210/41
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,424,118
App. No.
15/390,827
Granted
Sep 24, 2019
Kind
B2
Abstract

A method and a device are disclosed for the perspective representation, via an output image, of at least one virtual scene component arranged within a real scene. In the method, depth image data of the real scene is captured from a first perspective via a depth image sensor, and 2D image data of the real scene is captured from a second perspective via a 2D camera. Further, a virtual three-dimensional scene model of the real scene is created with reference to depth information from the depth image data, and at least one virtual scene component is inserted into the three-dimensional virtual model. Finally, an output image is generated by way of perspective projection of the 2D image data corresponding to the second perspective onto the virtual three-dimensional scene model comprising the virtual scene component.

Claims (33)

1. A method for perspective representation, via an output image, of at least one virtual scene component arranged within a real scene that includes an examination object, the method comprising:

capturing depth image data of the real scene including the examination object from a first perspective via a depth image sensor;

capturing 2D image data of the real scene including the examination object from a second perspective via a single 2D camera;

creating a virtual three-dimensional (3D) surface model of the real scene by converting depth information from the depth image data into the virtual 3D surface model such that the virtual 3D surface model describes a contour of a surface of the examination object, the depth information indicating a depth of the examination object;

inserting the at least one virtual scene component into the virtual 3D surface model at a location corresponding to a desired start point for medical imaging the examination object, the at least one virtual scene component being an object or element that is not contained in the real scene and is therefore not imaged via at least one of the depth image sensor and the single 2D camera, the inserting including (i) creating a geometric description of the at least one virtual scene component, the geometric description including a profile curve representing the depth of the examination object, and (ii) inserting the at least one virtual scene component into the virtual 3D surface model based on the profile curve such that a curvature of the at least one virtual scene component in the virtual 3D surface model corresponds to the depth of the examination object;

generating the output image using perspective projection to project the 2D image data corresponding to the second perspective along optical rays of the single 2D camera onto the virtual 3D surface model including the at least one virtual scene component such that the output image is a 2D image illustrating a perspectival correct version of the at least one virtual scene component; and

setting a scan region for the medical imaging of the examination object based on the perspectival correct version of the at least one virtual scene component included in the output image.

2. The method of claim 1 , wherein the perspective projection is effected via a renderer.

3. The method of claim 2 , wherein the perspective projection is effected with reference to at least one of extrinsic and intrinsic camera parameters of at least one of the single 2D camera and the depth image sensor.

4. The method of claim 3 , wherein the 2D image data and the depth image data are captured from a same perspective.

5. The method of claim 1 , wherein the perspective projection is effected with reference to at least one of extrinsic and intrinsic camera parameters of at least one of the single 2D camera and the depth image sensor.

6. The method of claim 5 , wherein the 2D image data and the depth image data are captured from a same perspective.

7. The method of claim 1 , wherein the real scene includes the examination object, situated on a patient couch of a medical imaging facility, and

the at least one virtual scene component corresponds to a scan region boundary.

8. The method of claim 7 , wherein the scan region boundary corresponds to a light marking or laser marking which is virtually projected onto the examination object.

9. The method of claim 7 , wherein the at least one virtual scene component corresponds to a boundary contour line as the scan region boundary.

10. The method of claim 1 , wherein the inserting of the at least one virtual scene component into the virtual 3D surface model is effected based on user inputs.

11. The method of claim 10 , wherein an arrangement of the at least one virtual scene component is determined using the 2D image data.

12. The method of claim 1 , wherein the 2D image data of the real scene is captured in an optical/visible wavelength range.

13. The method of claim 1 , wherein the depth image data of the real scene is captured in an infrared wavelength range.

14. A device for perspective representation, via an output image, of at least one virtual scene component arranged within a real scene that includes an examination object, the device comprising:

a depth image sensor, to capture depth image data of the real scene including the examination object from a first perspective;

a 2D camera to capture 2D image data of the real scene including the examination object from a second perspective; and

a memory and a processor, the memory containing computer readable code that, when executed by the processor, configures the processor to,

create a virtual three-dimensional (3D) surface model of the real scene by converting depth information from the depth image data into the virtual 3D surface model such that the virtual 3D surface model describes a contour of a surface of the examination object, the depth information indicating a depth of the examination object,

insert the at least one virtual scene component into the virtual 3D surface model at a location corresponding to a desired start point for medical imaging the examination object, the virtual scene component being an object or element that is not contained in the real scene and is therefore not imaged via at least one of the depth image sensor and the 2D camera, the processor configured to insert the virtual scene component by (i) creating a geometric description of the at least one virtual scene component, the geometric description including a profile curve representing the depth of the examination object, and (ii) inserting the at least one virtual scene component into the virtual 3D surface model based on the profile curve such that a curvature of the at least one virtual scene component in the virtual 3D surface model corresponds to the depth of the examination object,

generate the output image using perspective projection to project the 2D image data corresponding to the second perspective along optical rays of the 2D camera onto the virtual 3D surface model including the at least one virtual scene component such that the output image is a 2D image illustrating a perspectival correct version of the at least one virtual scene component, and

set a scan region for the medical imaging of the examination object based on the perspectival correct version of the at least one virtual scene component included in the output image.

15. The device of claim 14 , wherein the 2D camera and the depth image sensor are combined to form a camera unit.

16. A medical imaging facility, comprising the device of claim 14 .

17. The medical imaging facility of claim 16 , wherein the medical imaging facility is a computer tomography apparatus.

18. A medical imaging facility, comprising the device of claim 15 .

19. The medical imaging facility of claim 18 , wherein the medical imaging facility is a computer tomography apparatus.

Assignments (6)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Dec 20, 2023
From: SIEMENS HEALTHCARE GMBH
To: SIEMENS HEALTHINEERS AG
Reel/Frame 066267/0346 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Apr 19, 2017
From: SIEMENS AKTIENGESELLSCHAFT
To: SIEMENS HEALTHCARE GMBH
Reel/Frame 042055/0096 →
CORRECTIVE ASSIGNMENT TO CORRECT THE ASSIGNOR: WEIS, MICHAEL PREVIOUSLY RECORDED ON REEL 041483 FRAME 0880. ASSIGNOR(S) HEREBY CONFIRMS THE ASSIGNOR: WELS, MICHAEL. Recorded Mar 15, 2017
From: HANNEMANN, THILO; SUEHLING, MICHAEL; WELS, MICHAEL; WIMMER, ANDREAS
To: SIEMENS AKTIENGESELLSCHAFT
Reel/Frame 042011/0461 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Mar 13, 2017
From: ISO SOFTWARE SYSTEME GMBH
To: SIEMENS AKTIENGESELLSCHAFT
Reel/Frame 041552/0355 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Mar 7, 2017
From: DISTLER, FERDINAND
To: ISO SOFTWARE SYSTEME GMBH
Reel/Frame 041483/0854 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Mar 7, 2017
From: HANNEMANN, THILO; SUEHLING, MICHAEL; WEIS, MICHAEL; WIMMER, ANDREAS
To: SIEMENS AKTIENGESELLSCHAFT
Reel/Frame 041483/0880 →
Priority Claims (1)
DE 10 2016 200 225 · Jan 12, 2016 · national
Continuity (1)
Related Publication 20170200317A1 · Jul 13, 2017
Cited By (3)
US 12,261,988 US 12,299,907 US 12,664,668